<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>https://jtarchie.com</title>
    <link>https://jtarchie.com</link>
    <description>feed for https://jtarchie.com</description>
    <pubDate>Thu, 29 Oct 2026 02:07:37 +0000</pubDate>
    <item>
      <title>Writing</title>
      <link>https://jtarchie.com/posts/index.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="writing"><a href="#writing" rel="nofollow">#</a> Writing</h1>
<p>A note on my writing style. I write as I speak.</p>
<p>Newest first.</p>

<ul>
<li><time datetime="2025-01-26">2025-01-26</time> <a href="/posts/2025-01-26-efficient-api-design-avoiding-costly-external-requests.html" rel="nofollow">Efficient API Design: Avoiding Costly External Requests</a></li>
<li><time datetime="2024-08-30">2024-08-30</time> <a href="/posts/2024-08-30-exploring-goja-a-golang-javascript-runtime.html" rel="nofollow">Exploring Goja: A Golang JavaScript Runtime</a></li>
<li><time datetime="2024-07-02">2024-07-02</time> <a href="/posts/2024-07-02-optimizing-large-scale-openstreetmap-data-with-sqlite.html" rel="nofollow">Optimizing Large-Scale OpenStreetMap Data with SQLite</a></li>
<li><time datetime="2023-11-15">2023-11-15</time> <a href="/posts/2023-11-15-deploying-a-rails-application-with-aws-copilot.html" rel="nofollow">Deploying a Rails Application with AWS Copilot</a></li>
<li><time datetime="2023-10-30">2023-10-30</time> <a href="/posts/2023-10-30-implementing-a-simple-mutex-in-go.html" rel="nofollow">Implementing a simple Mutex in Go</a></li>
<li><time datetime="2023-10-14">2023-10-14</time> <a href="/posts/2023-10-14-transitioning-back-into-the-labor-force.html" rel="nofollow">Transitioning Back into the Labor Force</a></li>
<li><time datetime="2023-10-08">2023-10-08</time> <a href="/posts/2023-10-08-taking-time-off-between-jobs.html" rel="nofollow">Taking Time Off Between Jobs</a></li>
<li><time datetime="2023-09-30">2023-09-30</time> <a href="/posts/2023-09-30-my-preferred-go-stack.html" rel="nofollow">My Preferred Go Stack</a></li>
<li><time datetime="2023-09-21">2023-09-21</time> <a href="/posts/2023-09-21-seeking-optimal-sql-patterns-with-go.html" rel="nofollow">Seeking Optimal SQL Patterns with Go</a></li>
<li><time datetime="2023-09-20">2023-09-20</time> <a href="/posts/2023-09-20-exploring-go-with-cgo-overhead-and-sqlc.html" rel="nofollow">Exploring Go with CGO Overhead and `sqlc`</a></li>
<li><time datetime="2023-06-14">2023-06-14</time> <a href="/posts/2023-06-14-using-go-generics-for-function-caching.html" rel="nofollow">Using Go Generics for Function Caching</a></li>
<li><time datetime="2023-05-12">2023-05-12</time> <a href="/posts/2023-05-12-balancing-website-with-limited-access-hours.html" rel="nofollow">Balancing Website with Limited Access Hours</a></li>
<li><time datetime="2023-04-13">2023-04-13</time> <a href="/posts/2023-04-13-communication-process-tools.html" rel="nofollow">Communication &gt;&gt; Process &gt;&gt; Tools</a></li>
<li><time datetime="2023-04-06">2023-04-06</time> <a href="/posts/2023-04-06-programming-journey.html" rel="nofollow">Programming Journey</a></li>
<li><time datetime="2023-03-29">2023-03-29</time> <a href="/posts/2023-03-29-debugging-and-resolving-issues-with-go-and-the-sqlite3-library.html" rel="nofollow">Debugging and Resolving Issues with Go and the `sqlite3` Library</a></li>
<li><time datetime="2023-03-21">2023-03-21</time> <a href="/posts/2023-03-21-steps-toward-automation.html" rel="nofollow">Steps toward automation</a></li>
<li><time datetime="2023-03-05">2023-03-05</time> <a href="/posts/2023-03-05-programming-language-over-data-language.html" rel="nofollow">Programming Language over Data language</a></li>
<li><time datetime="2022-02-17">2022-02-17</time> <a href="/posts/2022-02-17-testing-thoughts.html" rel="nofollow">Testing Thoughts</a></li>
<li><time datetime="2022-01-03">2022-01-03</time> <a href="/posts/2022-01-03-when-in-san-diego.html" rel="nofollow">When in San Diego</a></li>
<li><time datetime="2021-08-06">2021-08-06</time> <a href="/posts/2021-08-06-embracing-tech-debt-as-a-team.html" rel="nofollow">Embracing Tech Debt as a Team</a></li>
<li><time datetime="2020-02-11">2020-02-11</time> <a href="/posts/2020-02-11-exploring-jit-execution-of-luajit.html" rel="nofollow">Exploring JIT execution of `luajit`</a></li>
<li><time datetime="2019-01-21">2019-01-21</time> <a href="/posts/2019-01-21-hello-world.html" rel="nofollow">Hello, World!</a></li>
</ul>

]]></content:encoded>
      <guid>https://jtarchie.com/posts/index.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Efficient API Design: Avoiding Costly External Requests</title>
      <link>https://jtarchie.com/posts/2025-01-26.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="efficient-api-design-avoiding-costly-external-requests"><a href="#efficient-api-design-avoiding-costly-external-requests" rel="nofollow">#</a> Efficient API Design: Avoiding Costly External Requests</h1>
<aside>Previous article on <a href="/posts/2024-08-30" rel="nofollow">using Goja</a> provides context and underlying technical details.</aside>
<p>On a recent project, I built an API that leverages OpenStreetMap data, currently
focusing on the United States and Canada to search for points of interest near
each other. The API endpoint for querying that data ran into problems of slow
response times, both for querying and returning considerable data (megabyte
payloads), with most of it often discarded on the client side. This post goes
through the optimization process to have the user experience go from tens of
seconds to a sub-second experience.</p>
<p>Initially, my front-end implementation included an endpoint that allowed users
to query OpenStreetMap data in a manner similar to other OpenStreetMap systems.
The queries focused on nodes, ways, or relations and their associated tags.</p>
<p>For example, to find all the Starbucks locations in Colorado, you could use the
following endpoint and query parameter:</p>
<ul>
<li><strong>Endpoint:</strong> <code>/api/search</code></li>
<li><strong>Query parameter:</strong> <code>search=nwr[name=Starbucks](area=colorado)</code></li>
</ul>
<p>The response would return a GeoJSON FeatureCollection, as shown below:</p>
<pre><code><span><span><span>{</span>
</span></span><span><span>  <span>&#34;type&#34;</span><span>:</span> <span>&#34;FeatureCollection&#34;</span><span>,</span>
</span></span><span><span>  <span>&#34;features&#34;</span><span>:</span> <span>[</span>
</span></span><span><span>    <span>{</span>
</span></span><span><span>      <span>&#34;type&#34;</span><span>:</span> <span>&#34;Feature&#34;</span><span>,</span>
</span></span><span><span>      <span>&#34;geometry&#34;</span><span>:</span> <span>{</span>
</span></span><span><span>        <span>&#34;type&#34;</span><span>:</span> <span>&#34;Point&#34;</span><span>,</span>
</span></span><span><span>        <span>&#34;coordinates&#34;</span><span>:</span> <span>[</span><span>125.6</span><span>,</span> <span>10.1</span><span>]</span>
</span></span><span><span>      <span>},</span>
</span></span><span><span>      <span>&#34;properties&#34;</span><span>:</span> <span>{</span>
</span></span><span><span>        <span>&#34;name&#34;</span><span>:</span> <span>&#34;Starbucks&#34;</span>
</span></span><span><span>      <span>}</span>
</span></span><span><span>    <span>}</span>
</span></span><span><span>  <span>]</span>
</span></span><span><span><span>}</span>
</span></span></code></pre><p>While this worked, the objective was to allow correlations between entities,
such as identifying schools near coffee shops or places not within a certain
distance of each other. I used TurfJS for these operations in the first
iteration.</p>
<pre><code><span><span><span>import</span> <span>turf</span> <span>from</span> <span>&#34;@turf/turf&#34;</span><span>;</span>
</span></span><span><span><span>const</span> <span>coffeeShops</span> <span>=</span> <span>await</span> <span>query</span><span>(</span><span>`nwr[amenity=cafe][name](area=colorado)`</span><span>);</span>
</span></span><span><span><span>const</span> <span>highSchools</span> <span>=</span> <span>await</span> <span>query</span><span>(</span><span>`nwr[amenity=school][name](area=colorado)`</span><span>);</span>
</span></span><span><span>
</span></span><span><span><span>const</span> <span>nearbySchools</span> <span>=</span> <span>[];</span>
</span></span><span><span><span>coffeeShops</span><span>.</span><span>features</span><span>.</span><span>forEach</span><span>((</span><span>coffeeShop</span><span>)</span> <span>=&gt;</span> <span>{</span>
</span></span><span><span>  <span>highSchools</span><span>.</span><span>features</span><span>.</span><span>forEach</span><span>((</span><span>school</span><span>)</span> <span>=&gt;</span> <span>{</span>
</span></span><span><span>    <span>const</span> <span>distance</span> <span>=</span> <span>turf</span><span>.</span><span>distance</span><span>(</span><span>coffeeShop</span><span>,</span> <span>school</span><span>,</span> <span>{</span> <span>units</span><span>:</span> <span>&#34;kilometers&#34;</span> <span>});</span>
</span></span><span><span>    <span>if</span> <span>(</span><span>distance</span> <span>&lt;</span> <span>1</span><span>)</span> <span>{</span> <span>// assuming 1 kilometer as the proximity threshold
</span></span></span><span><span><span></span>      <span>nearbySchools</span><span>.</span><span>push</span><span>({</span>
</span></span><span><span>        <span>coffeeShop</span><span>:</span> <span>coffeeShop</span><span>.</span><span>properties</span><span>.</span><span>name</span><span>,</span>
</span></span><span><span>        <span>school</span><span>:</span> <span>school</span><span>.</span><span>properties</span><span>.</span><span>name</span><span>,</span>
</span></span><span><span>        <span>distance</span><span>:</span> <span>distance</span><span>,</span>
</span></span><span><span>      <span>});</span>
</span></span><span><span>    <span>}</span>
</span></span><span><span>  <span>});</span>
</span></span><span><span><span>});</span>
</span></span></code></pre><p>The initial approach was functional but slow. For example, querying for schools
in a large state like California could return thousands of results. Transferring
this data from the server to the client and performing geospatial analyses in
the browser with TurfJS was inefficient. Queries often took between 5 and 10
seconds, depending on complexity.</p>
<p>The request-response cycle looked like this:</p>
<pre>%%{init: {&#39;theme&#39;:&#39;forest&#39;}}%%
sequenceDiagram
    participant Client
    participant Server
    Client-&gt;&gt;Server: GET /api/search?search=nwr[amenity=cafe][name](area=colorado)
    Note right of Server: Returns large GeoJSON&lt;br&gt;FeatureCollection for coffeeShops
    Server--&gt;&gt;Client: GeoJSON (size: ~500KB)
    Client-&gt;&gt;Server: GET /api/search?search=nwr[amenity=school][name](area=colorado)
    Note right of Server: Returns large GeoJSON&lt;br&gt;FeatureCollection for highSchools
    Server--&gt;&gt;Client: GeoJSON (size: ~1MB)
    Note left of Client: Client processes data&lt;br&gt;with TurfJS
    Client-&gt;&gt;Client: Calculate distances and filter
    Note left of Client: Result: nearbySchools
</pre><p>While this was acceptable for a prototype, I wanted to improve performance. The
idea of moving correlation and geospatial analysis to the server side intrigued
me, inspired by conversations with a former coworker who extended their Go
runtime to support scripting. This led to an exploration of embedding JavaScript
execution on the server using <a href="https://github.com/dop251/goja" rel="nofollow">Goja</a>.</p>
<p>Goja is a lightweight JavaScript interpreter written in Go.
<a href="/posts/2024-08-30-exploring-goja-a-golang-javascript-runtime" rel="nofollow">Embedding it into my project was straightforward</a>,
allowing me to expose application infrastructure for server-side JavaScript
execution. This approach let users write JavaScript code to query and correlate
data without sending large datasets over the wire. Instead, the server executed
these queries in a sandboxed environment and returned only the necessary
results.</p>
<p>Initially, I attempted to reuse the same client-side logic, including TurfJS, on
the server. However, TurfJS, being computationally intensive, struggled within
Goja’s interpreted environment, leading to query times of 15 to 20 seconds.
Clearly, this was not a viable solution.</p>
<p>To address this, I offloaded the heavy computations to optimized Go functions.
My project already used the Orb library for geospatial operations, so I exposed
Orb’s functionality to Goja. This allowed JavaScript code to call underlying Go
functions for tasks like clustering and distance calculations. The result was a
dramatic improvement in performance, reducing query times to sub-second ranges
while minimizing data transfer.</p>
<ul>
<li><strong>Endpoint:</strong> <code>/api/runtime</code></li>
<li><strong>Query parameter:</strong> <code>source=&lt;url encoded JavaScript file&gt;</code></li>
</ul>
<pre><code><span><span><span>const</span> <span>coffeeShops</span> <span>=</span> <span>query</span><span>.</span><span>execute</span><span>(</span><span>`nwr[amenity=cafe][name](area=colorado)`</span><span>);</span>
</span></span><span><span><span>const</span> <span>highSchools</span> <span>=</span> <span>query</span><span>.</span><span>execute</span><span>(</span><span>`nwr[amenity=school][name](area=colorado)`</span><span>);</span>
</span></span><span><span>
</span></span><span><span><span>const</span> <span>clusteredShops</span> <span>=</span> <span>coffeeShops</span><span>.</span><span>cluster</span><span>(</span><span>10</span><span>);</span> <span>// find coffee shops within 10m of each other
</span></span></span><span><span><span></span>
</span></span><span><span><span>const</span> <span>results</span> <span>=</span> <span>clusteredShops</span><span>.</span><span>flatMap</span><span>((</span><span>shop</span><span>)</span> <span>=&gt;</span> <span>{</span>
</span></span><span><span>  <span>// find nearby high schools within 1km, returning at most 1 entry
</span></span></span><span><span><span></span>  <span>const</span> <span>nearbyHighSchools</span> <span>=</span> <span>clusteredShops</span><span>.</span><span>overlap</span><span>(</span><span>highSchools</span><span>,</span> <span>1_000</span><span>,</span> <span>0</span><span>,</span> <span>1</span><span>);</span>
</span></span><span><span>  <span>return</span> <span>[</span><span>shop</span><span>,</span> <span>nearbyHighSchools</span><span>];</span>
</span></span><span><span><span>});</span>
</span></span><span><span>
</span></span><span><span><span>const</span> <span>payload</span> <span>=</span> <span>results</span><span>.</span><span>asGeoJSON</span><span>();</span>
</span></span><span><span>
</span></span><span><span><span>export</span> <span>{</span> <span>payload</span> <span>};</span>
</span></span></code></pre><p>The system now supports submitting JavaScript code as a parameter to an API
endpoint. This code runs in a sandboxed environment with strict timeouts and
safely interacts with the underlying data. The data remains read-only, and the
platform operates entirely on ephemeral infrastructure hosted by me.</p>
<p>The current platform enables users to test and experiment with running
JavaScript in a sandboxed environment for geospatial queries. It draws parallels
to Overpass QL (OQL) used in OpenStreetMap’s Turbo, but I find JavaScript easier
to reason with, especially with TypeScript support. While the API transpiles
TypeScript to JavaScript without type checking, it includes a TypeScript
definition for the sandbox environment. This combination provides a flexible and
user-friendly way to explore geospatial data.</p>
<p>There is more information in the <a href="https://knowhere.live/docs" rel="nofollow">documentation</a> for
this project. It should be open to more people soon.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2025-01-26.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Exploring Goja: A Golang JavaScript Runtime</title>
      <link>https://jtarchie.com/posts/2024-08-30.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="exploring-goja-a-golang-javascript-runtime"><a href="#exploring-goja-a-golang-javascript-runtime" rel="nofollow">#</a> Exploring Goja: A Golang JavaScript Runtime</h1>
<p>This post explores <a href="https://github.com/dop251/goja" rel="nofollow">Goja</a>, a JavaScript runtime
library in the Golang ecosystem. Goja stands out as a powerful tool for
embedding JavaScript within Go applications, offering unique advantages when
manipulating data and exposing an SDK that doesn’t require a <code>go build</code> step.</p>
<h2 id="background-the-need-for-goja"><a href="#background-the-need-for-goja" rel="nofollow">#</a> Background: The Need for Goja</h2>
<p>In my project, I encountered challenges when querying and manipulating large
datasets. Initially, everything was written in Go, which was efficient but
became cumbersome, especially when dealing with complex JSON responses. While
Go’s minimalistic approach is generally advantageous, the verbosity required for
specific tasks slowed me down.</p>
<p>Using an embedded scripting language could simplify the process, leading me to
explore various options. Lua was my first choice because of its reputation for
being lightweight and embeddable. Still, I quickly found that the available Go
libraries for Lua were all over the place in implementations, versions (5.1,
5.2, etc), and active support.</p>
<p>I then investigated other popular scripting languages in the Go ecosystem. I
considered options like <a href="https://github.com/expr-lang/expr/" rel="nofollow">Expr</a>,
<a href="https://github.com/tommie/v8go" rel="nofollow">V8</a>, and
<a href="https://github.com/google/starlark-go" rel="nofollow">Starlark</a>, but eventually, Goja emerged
as the most promising candidate.</p>
<p>Here is the
<a href="https://github.com/jtarchie/benchmark-tests/blob/22789057b4fcf95443ea8cb61f261dea31935cda/eval_benchmark_test.go" rel="nofollow">GitHub repository</a>,
where I conducted some benchmarks on these libraries, testing their performance
and ease of integration with Go.</p>
<h2 id="why-goja"><a href="#why-goja" rel="nofollow">#</a> Why Goja?</h2>
<p>Goja won me over because of its seamless integration with Go structs. When you
assign a Go struct to a value in the JavaScript runtime, Goja automatically
infers the fields and methods, making them accessible in JavaScript without
requiring a separate bridge layer. It leverages Go’s reflection capabilities to
invoke getters and setters on these fields, offering a robust and transparent
interaction between Go and JavaScript.</p>
<p>Let’s dive into some examples to see Goja in action. These examples highlight
features I’ve found useful, but desired to have more examples in the
documentation with.</p>
<h2 id="assigning-and-returning-values"><a href="#assigning-and-returning-values" rel="nofollow">#</a> Assigning and Returning Values</h2>
<p>To start, let’s take a simple example where we pass an array of integers from Go
to the JavaScript runtime and filter out the even values.</p>
<pre><code><span><span><span>package</span> <span>main</span>
</span></span><span><span>
</span></span><span><span><span>import</span> <span>(</span>
</span></span><span><span>	<span>&#34;fmt&#34;</span>
</span></span><span><span>
</span></span><span><span>	<span>&#34;github.com/dop251/goja&#34;</span>
</span></span><span><span><span>)</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>main</span><span>()</span> <span>{</span>
</span></span><span><span>	<span>vm</span> <span>:=</span> <span>goja</span><span>.</span><span>New</span><span>()</span>
</span></span><span><span>
</span></span><span><span>	<span>// Passing an array of integers from 1 to 100
</span></span></span><span><span><span></span>	<span>values</span> <span>:=</span> <span>[]</span><span>int</span><span>{}</span>
</span></span><span><span>	<span>for</span> <span>i</span> <span>:=</span> <span>1</span><span>;</span> <span>i</span> <span>&lt;=</span> <span>100</span><span>;</span> <span>i</span><span>++</span> <span>{</span>
</span></span><span><span>		<span>values</span> <span>=</span> <span>append</span><span>(</span><span>values</span><span>,</span> <span>i</span><span>)</span>
</span></span><span><span>	<span>}</span>
</span></span><span><span>
</span></span><span><span>	<span>// Define the JavaScript code to filter even values
</span></span></span><span><span><span></span>	<span>script</span> <span>:=</span> <span>`
</span></span></span><span><span><span>		values.filter((x) =&gt; {
</span></span></span><span><span><span>			return x % 2 === 0;
</span></span></span><span><span><span>		})
</span></span></span><span><span><span>  `</span>
</span></span><span><span>
</span></span><span><span>	<span>// Set the array in the JavaScript runtime
</span></span></span><span><span><span></span>	<span>vm</span><span>.</span><span>Set</span><span>(</span><span>&#34;values&#34;</span><span>,</span> <span>values</span><span>)</span>
</span></span><span><span>
</span></span><span><span>	<span>// Run the script
</span></span></span><span><span><span></span>	<span>result</span><span>,</span> <span>err</span> <span>:=</span> <span>vm</span><span>.</span><span>RunString</span><span>(</span><span>script</span><span>)</span>
</span></span><span><span>	<span>if</span> <span>err</span> <span>!=</span> <span>nil</span> <span>{</span>
</span></span><span><span>		<span>panic</span><span>(</span><span>err</span><span>)</span>
</span></span><span><span>	<span>}</span>
</span></span><span><span>
</span></span><span><span>	<span>// Convert the result back to a Go slice of empty interfaces
</span></span></span><span><span><span></span>	<span>filteredValues</span> <span>:=</span> <span>result</span><span>.</span><span>Export</span><span>().([]</span><span>interface</span><span>{})</span>
</span></span><span><span>
</span></span><span><span>	<span>fmt</span><span>.</span><span>Println</span><span>(</span><span>filteredValues</span><span>)</span>
</span></span><span><span>	<span>// Outputs: [2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 100]
</span></span></span><span><span><span></span>
</span></span><span><span>	<span>first</span> <span>:=</span> <span>filteredValues</span><span>[</span><span>0</span><span>].(</span><span>int64</span><span>)</span>
</span></span><span><span>	<span>fmt</span><span>.</span><span>Println</span><span>(</span><span>first</span><span>)</span>
</span></span><span><span><span>}</span>
</span></span></code></pre><p>In this example, you can see that iterating through an array in Goja doesn’t
require explicit type annotations. Goja is able to infer the type of the array
based on its content, thanks to Go’s reflection mechanism. When filtering the
values and returning the result, Goja converts the result back to an array of
empty interfaces (<code>[]interface{}</code>). This is because Goja needs to handle
JavaScript’s dynamic typing within Go’s static type system.</p>
<p>If you need to work with the resulting values in Go, you’ll have to perform type
assertions to extract the integers. Internally, Goja represents all integers as
<code>int64</code>.</p>
<h2 id="structs-and-method-calls"><a href="#structs-and-method-calls" rel="nofollow">#</a> Structs and Method Calls</h2>
<p>Next, let’s explore how Goja handles Go structs, particularly focusing on
methods and exported fields.</p>
<pre><code><span><span><span>package</span> <span>main</span>
</span></span><span><span>
</span></span><span><span><span>import</span> <span>(</span>
</span></span><span><span>    <span>&#34;fmt&#34;</span>
</span></span><span><span>    <span>&#34;github.com/dop251/goja&#34;</span>
</span></span><span><span><span>)</span>
</span></span><span><span>
</span></span><span><span><span>type</span> <span>Person</span> <span>struct</span> <span>{</span>
</span></span><span><span>    <span>Name</span> <span>string</span>
</span></span><span><span>    <span>age</span>  <span>int</span>
</span></span><span><span><span>}</span>
</span></span><span><span>
</span></span><span><span><span>// Method to get the age (unexported)
</span></span></span><span><span><span></span><span>func</span> <span>(</span><span>p</span> <span>*</span><span>Person</span><span>)</span> <span>GetAge</span><span>()</span> <span>int</span> <span>{</span>
</span></span><span><span>    <span>return</span> <span>p</span><span>.</span><span>age</span>
</span></span><span><span><span>}</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>main</span><span>()</span> <span>{</span>
</span></span><span><span>    <span>vm</span> <span>:=</span> <span>goja</span><span>.</span><span>New</span><span>()</span>
</span></span><span><span>
</span></span><span><span>    <span>// Create a new Person instance
</span></span></span><span><span><span></span>    <span>person</span> <span>:=</span> <span>&amp;</span><span>Person</span><span>{</span>
</span></span><span><span>        <span>Name</span><span>:</span> <span>&#34;John Doe&#34;</span><span>,</span>
</span></span><span><span>        <span>age</span><span>:</span>  <span>30</span><span>,</span>
</span></span><span><span>    <span>}</span>
</span></span><span><span>
</span></span><span><span>    <span>// Set the Person struct in the JavaScript runtime
</span></span></span><span><span><span></span>    <span>vm</span><span>.</span><span>Set</span><span>(</span><span>&#34;person&#34;</span><span>,</span> <span>person</span><span>)</span>
</span></span><span><span>
</span></span><span><span>    <span>// JavaScript code to access the struct&#39;s fields and methods
</span></span></span><span><span><span></span>    <span>script</span> <span>:=</span> <span>`
</span></span></span><span><span><span>        const name = person.Name;    // Access exported field
</span></span></span><span><span><span>        const age = person.GetAge(); // Access unexported field via getter
</span></span></span><span><span><span>        name + &#34; is &#34; + age + &#34; years old.&#34;;
</span></span></span><span><span><span>    `</span>
</span></span><span><span>
</span></span><span><span>    <span>result</span><span>,</span> <span>err</span> <span>:=</span> <span>vm</span><span>.</span><span>RunString</span><span>(</span><span>script</span><span>)</span>
</span></span><span><span>    <span>if</span> <span>err</span> <span>!=</span> <span>nil</span> <span>{</span>
</span></span><span><span>        <span>panic</span><span>(</span><span>err</span><span>)</span>
</span></span><span><span>    <span>}</span>
</span></span><span><span>
</span></span><span><span>    <span>fmt</span><span>.</span><span>Println</span><span>(</span><span>result</span><span>.</span><span>String</span><span>())</span> <span>// Outputs: John Doe is 30 years old.
</span></span></span><span><span><span></span><span>}</span>
</span></span></code></pre><p>In this example, I’ve defined a <code>Person</code> struct with an exported <code>Name</code> field
and an unexported <code>age</code> field. The <code>GetName</code> method is exported. When accessing
these fields and methods from JavaScript, Goja adheres to the naming conventions
on the struct. The method <code>GetAge</code> is accessed as <code>GetName</code>.</p>
<p>There is a pattern for making the Javascript naming convention of camel case
translate to Golang naming convention via
<a href="https://pkg.go.dev/github.com/dop251/goja#FieldNameMapper" rel="nofollow"><code>FieldNameMapper</code></a>.
This allows for the Go method <code>GetAge</code> to be called as <code>getAge</code> in the
javascript invocation.</p>
<h2 id="exception-handling"><a href="#exception-handling" rel="nofollow">#</a> Exception Handling</h2>
<p>When an exception occurs in JavaScript, Goja uses standard Go error handling to
manage it. Let’s explore an example of a runtime exception—division by zero.</p>
<pre><code><span><span><span>package</span> <span>main</span>
</span></span><span><span>
</span></span><span><span><span>import</span> <span>(</span>
</span></span><span><span>	<span>&#34;errors&#34;</span>
</span></span><span><span>	<span>&#34;fmt&#34;</span>
</span></span><span><span>
</span></span><span><span>	<span>&#34;github.com/dop251/goja&#34;</span>
</span></span><span><span><span>)</span>
</span></span><span><span>
</span></span><span><span><span>// JavaScript code that triggers a division by zero error
</span></span></span><span><span><span></span><span>const</span> <span>script</span> <span>=</span> <span>`
</span></span></span><span><span><span>	// Using BigInt notation in JavaScript
</span></span></span><span><span><span>	const a = 1n / 0n;
</span></span></span><span><span><span>`</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>main</span><span>()</span> <span>{</span>
</span></span><span><span>	<span>vm</span> <span>:=</span> <span>goja</span><span>.</span><span>New</span><span>()</span>
</span></span><span><span>
</span></span><span><span>	<span>// Execute the JavaScript code
</span></span></span><span><span><span></span>	<span>_</span><span>,</span> <span>err</span> <span>:=</span> <span>vm</span><span>.</span><span>RunString</span><span>(</span><span>script</span><span>)</span>
</span></span><span><span>
</span></span><span><span>	<span>// Handle any errors that occur
</span></span></span><span><span><span></span>	<span>var</span> <span>exception</span> <span>*</span><span>goja</span><span>.</span><span>Exception</span>
</span></span><span><span>	<span>if</span> <span>errors</span><span>.</span><span>As</span><span>(</span><span>err</span><span>,</span> <span>&amp;</span><span>exception</span><span>)</span> <span>{</span>
</span></span><span><span>		<span>fmt</span><span>.</span><span>Printf</span><span>(</span><span>&#34;JavaScript error: %s\n&#34;</span><span>,</span> <span>exception</span><span>.</span><span>Error</span><span>())</span>
</span></span><span><span>		<span>// Output: JavaScript error: RangeError: Division by zero at &lt;eval&gt;:1:1(3)
</span></span></span><span><span><span></span>	<span>}</span> <span>else</span> <span>if</span> <span>err</span> <span>!=</span> <span>nil</span> <span>{</span>
</span></span><span><span>		<span>// Handle other types of errors (if any)
</span></span></span><span><span><span></span>		<span>fmt</span><span>.</span><span>Printf</span><span>(</span><span>&#34;Error: %s\n&#34;</span><span>,</span> <span>err</span><span>.</span><span>Error</span><span>())</span>
</span></span><span><span>	<span>}</span>
</span></span><span><span><span>}</span>
</span></span></code></pre><p>The error value returned is of type <code>*goja.Exception</code>, which provides
information about the JavaScript exception that was raised and where it failed.
While I haven’t found a strong need to inspect these errors beyond logging them
to services like New Relic or DataDog, Goja does offer the tools to do so if
necessary.</p>
<p>Additionally, Goja can raise other types of exceptions, such as
<code>*goja.StackOverflowError</code>, <code>*goja.InterruptedError</code>, and
<code>*goja.CompilerSyntaxError</code>, which correspond to specific issues related to the
interpreter. These exceptions can be useful to handle and report, especially
when dealing with clients that execute JavaScript code.</p>
<h2 id="sandbox-user-code-with-a-pool-of-vms"><a href="#sandbox-user-code-with-a-pool-of-vms" rel="nofollow">#</a> Sandbox User Code with a Pool of VMs</h2>
<p>While developing my application, I noticed that initializing the VM took a
significant amount of time. Each VM required global modules that needed to be
available to the user at runtime. Go provides <code>sync.Pool</code> to help <em>reuse</em>
objects, making it a perfect fit for my use case to avoid the overhead of heavy
initialization.</p>
<p>Here’s an example of a pool of Goja VMs:</p>
<pre><code><span><span><span>package</span> <span>main</span>
</span></span><span><span>
</span></span><span><span><span>import</span> <span>(</span>
</span></span><span><span>	<span>&#34;fmt&#34;</span>
</span></span><span><span>	<span>&#34;sync&#34;</span>
</span></span><span><span>
</span></span><span><span>	<span>&#34;github.com/dop251/goja&#34;</span>
</span></span><span><span><span>)</span>
</span></span><span><span>
</span></span><span><span><span>var</span> <span>vmPool</span> <span>=</span> <span>sync</span><span>.</span><span>Pool</span><span>{</span>
</span></span><span><span>	<span>New</span><span>:</span> <span>func</span><span>()</span> <span>interface</span><span>{}</span> <span>{</span>
</span></span><span><span>		<span>vm</span> <span>:=</span> <span>goja</span><span>.</span><span>New</span><span>()</span>
</span></span><span><span>
</span></span><span><span>		<span>// Define a global function available in every VM
</span></span></span><span><span><span></span>		<span>vm</span><span>.</span><span>Set</span><span>(</span><span>&#34;add&#34;</span><span>,</span> <span>func</span><span>(</span><span>a</span><span>,</span> <span>b</span> <span>int</span><span>)</span> <span>int</span> <span>{</span>
</span></span><span><span>			<span>return</span> <span>a</span> <span>+</span> <span>b</span>
</span></span><span><span>		<span>})</span>
</span></span><span><span>
</span></span><span><span>		<span>// ... other global values set ...
</span></span></span><span><span><span></span>
</span></span><span><span>		<span>return</span> <span>vm</span>
</span></span><span><span>	<span>},</span>
</span></span><span><span><span>}</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>main</span><span>()</span> <span>{</span>
</span></span><span><span>	<span>vm</span> <span>:=</span> <span>vmPool</span><span>.</span><span>Get</span><span>().(</span><span>*</span><span>goja</span><span>.</span><span>Runtime</span><span>)</span>
</span></span><span><span>	<span>// Put the VM back into the pool for reuse
</span></span></span><span><span><span></span>	<span>defer</span> <span>vmPool</span><span>.</span><span>Put</span><span>(</span><span>vm</span><span>)</span>
</span></span><span><span>
</span></span><span><span>	<span>script</span> <span>:=</span> <span>`
</span></span></span><span><span><span>		const result = add(5, 10);
</span></span></span><span><span><span>		result;
</span></span></span><span><span><span>	`</span>
</span></span><span><span>
</span></span><span><span>	<span>value</span><span>,</span> <span>err</span> <span>:=</span> <span>vm</span><span>.</span><span>RunString</span><span>(</span><span>script</span><span>)</span>
</span></span><span><span>	<span>if</span> <span>err</span> <span>!=</span> <span>nil</span> <span>{</span>
</span></span><span><span>		<span>panic</span><span>(</span><span>err</span><span>)</span>
</span></span><span><span>	<span>}</span>
</span></span><span><span>
</span></span><span><span>	<span>fmt</span><span>.</span><span>Println</span><span>(</span><span>&#34;Result:&#34;</span><span>,</span> <span>value</span><span>.</span><span>Export</span><span>())</span>
</span></span><span><span>	<span>// Result: 15
</span></span></span><span><span><span></span><span>}</span>
</span></span></code></pre><p>Since <a href="https://pkg.go.dev/sync#Pool" rel="nofollow"><code>sync.Pool</code></a> is well-documented, let’s
focus on the JavaScript runtime. In this example, the user declares a variable
<code>result</code>, and its value is returned. However, we encounter a limitation: the VM
cannot be reused as is.</p>
<p>The global namespace has been polluted with the variable <code>result</code>. If I rerun
the same code with the same pool, I receive the following error:
<code>SyntaxError: Identifier &#39;result&#39; has already been declared at &lt;eval&gt;:1:1(0)</code>.
There is a <a href="https://github.com/dop251/goja/issues/205" rel="nofollow">GitHub issue</a> that
recommends clearing the value of <code>result</code> each time. However, I found this
pattern impractical due to the added complexity when dealing with user-provided
code.</p>
<p>Until now, the examples I’ve given have been demonstrations of predefined code.
My application, however, allows users to provide their own code to run within
the Goja runtime. This required some experimentation,
<a href="https://github.com/pocketbase/pocketbase/blob/5547c0deded8f9cc329cd6f0670aef19e2a3001a/plugins/jsvm/binds.go#L218" rel="nofollow">exploration</a>,
and the adoption of patterns to avoid the “already declared” error.</p>
<pre><code><span><span><span>value</span><span>,</span> <span>err</span> <span>:=</span> <span>vm</span><span>.</span><span>RunString</span><span>(</span><span>&#34;(function() {&#34;</span> <span>+</span> <span>userCode</span> <span>+</span> <span>&#34;})()&#34;</span><span>)</span>
</span></span><span><span><span>if</span> <span>err</span> <span>!=</span> <span>nil</span> <span>{</span>
</span></span><span><span>	<span>panic</span><span>(</span><span>err</span><span>)</span>
</span></span><span><span><span>}</span>
</span></span></code></pre><p>The final solution for sandboxing user code involves executing the <code>userCode</code>
within an anonymous function in its own scope. Since the function isn’t named,
it isn’t assigned globally, and therefore doesn’t require cleanup. After some
benchmark testing, I confirmed that garbage collection effectively cleans it up
as well.</p>
<p>Hopefully, my experience with the nuances of Goja helps you get started quickly!</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2024-08-30.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Optimizing Large-Scale OpenStreetMap Data with SQLite</title>
      <link>https://jtarchie.com/posts/2024-07-02.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="optimizing-large-scale-openstreetmap-data-with-sqlite"><a href="#optimizing-large-scale-openstreetmap-data-with-sqlite" rel="nofollow">#</a> Optimizing Large-Scale OpenStreetMap Data with SQLite</h1>
<aside><a href="https://news.ycombinator.com/item?id=40864091" rel="nofollow">Discuss on Hackernews</a></aside>
<p>Over the past year or two, I’ve worked on a project to convert a massive dataset
into an SQLite database. The original data was in a compressed binary format
known as OSMPBF, which stands for OpenStreetMap Protocol Buffer Format. This
format is highly compact and compressed, making it difficult to search. The goal
of converting it into an SQLite database was to leverage SQLite’s search
functionalities, such as full-text search, R-tree indexes, and traditional
B-tree indexes on database table columns.</p>
<p>The <a href="https://www.openstreetmap.org" rel="nofollow">OpenStreetMap (OSM)</a> data is categorized
into three main elements: nodes, ways, and relations. A node represents a single
latitude-longitude point, akin to a point along a trail. A way is a series of
nodes forming a path that can be a shape. A relation is an element that can
include other relations, ways, or nodes, such as an entire trail system. Each
component can have metadata associated with it, documented in a well-maintained
<a href="https://wiki.openstreetmap.org/wiki/Tags" rel="nofollow">OSM wiki</a>.</p>
<pre>%%{init: {&#39;theme&#39;:&#39;forest&#39;}}%%
graph TD
    subgraph Hiking Trail
        Node1(Node)
        Node2(Node)
        Node3(Node)
        Way(Way)
    end

    subgraph Mountain Bike Trail
        Node4(Node)
        Node5(Node)
        Way1(Way)
    end

    subgraph Trail System
        Relation
    end

    Node1 --&gt; Way
    Node2 --&gt; Way
    Node3 --&gt; Way
    Node4 --&gt; Way1
    Node5 --&gt; Way1
    Way1 --&gt; Relation
    Way --&gt; Relation
</pre><p>I’m using the Open Street Map data of the entire
<a href="https://download.geofabrik.de/north-america/us.html" rel="nofollow">United States</a> for
project. The stats of the file are around 1.4 billion entries, 9GB in size, and
tags and bounding boxes are deduplicated to save space.</p>
<p>My first task was to transfer this OSM data from its compressed file format into
SQLite. Given the inconsistent tagging across different elements, I used a JSON
data type for tags while keeping other consistent information, such as latitude,
longitude, and element type, in regular columns. This initial SQLite database
was enormous, around 100 gigabytes for the United States, which necessitated
determining which data was essential and how to optimize searches.</p>
<pre><code><span><span><span>CREATE</span><span> </span><span>TABLE</span><span> </span><span>entries</span><span> </span><span>(</span><span>
</span></span></span><span><span><span>	</span><span>id</span><span>       </span><span>INTEGER</span><span> </span><span>PRIMARY</span><span> </span><span>KEY</span><span> </span><span>AUTOINCREMENT</span><span>,</span><span>
</span></span></span><span><span><span>	</span><span>osm_id</span><span>   </span><span>INTEGER</span><span> </span><span>NOT</span><span> </span><span>NULL</span><span>,</span><span>
</span></span></span><span><span><span>	</span><span>osm_type</span><span> </span><span>INTEGER</span><span> </span><span>NOT</span><span> </span><span>NULL</span><span>,</span><span>
</span></span></span><span><span><span>	</span><span>minLat</span><span>   </span><span>REAL</span><span>,</span><span>
</span></span></span><span><span><span>	</span><span>maxLat</span><span>   </span><span>REAL</span><span>,</span><span>
</span></span></span><span><span><span>	</span><span>minLon</span><span>   </span><span>REAL</span><span>,</span><span>
</span></span></span><span><span><span>	</span><span>maxLon</span><span>   </span><span>REAL</span><span>,</span><span>
</span></span></span><span><span><span>	</span><span>tags</span><span>     </span><span>BLOB</span><span>,</span><span> </span><span>-- key-value pair of tags (JSON)
</span></span></span><span><span><span></span><span>	</span><span>refs</span><span>     </span><span>BLOB</span><span>  </span><span>-- array of nodes, ways, and relations
</span></span></span><span><span><span></span><span>)</span><span> </span><span>STRICT</span><span>;</span><span>
</span></span></span></code></pre><p>For instance, a query like “Find all the Costcos” would be practical, but due to
the vast dataset, running a query took over a minute. I realized I needed to
process the data further. By filtering down to elements with specific tags like
name, shop type, and amenity, I reduced the database size to about 40 gigabytes.
Although searches became faster, they were still too slow for practical use,
often taking tens of seconds.</p>
<p>To improve query performance, I explored SQLite’s indexing capabilities. While
SQLite doesn’t support the same JSON indexing as Postgres, I could create
indexes for individual tags within the JSON.</p>
<pre><code><span><span><span>CREATE</span><span> </span><span>INDEX</span><span> </span><span>entries_name</span><span> </span><span>ON</span><span> </span><span>entries</span><span>(</span><span>tags</span><span>-&gt;&gt;</span><span>&#39;name&#39;</span><span>);</span><span>
</span></span></span></code></pre><p>However, this requires an index per tag, which won’t scale, especially for a
dynamic list of tags. SQLite does offer full-text search for unstructured text,
such as a document. I adapted this by concatenating JSON keys and values into a
single string for full-text indexing, using the following SQL:</p>
<pre><code><span><span><span>CREATE</span><span> </span><span>VIRTUAL</span><span> </span><span>TABLE</span><span> </span><span>search</span><span> </span><span>USING</span><span> </span><span>fts5</span><span>(</span><span>tags</span><span>);</span><span>
</span></span></span><span><span><span>
</span></span></span><span><span><span></span><span>WITH</span><span> </span><span>tags</span><span> </span><span>AS</span><span> </span><span>(</span><span>
</span></span></span><span><span><span>	</span><span>SELECT</span><span>
</span></span></span><span><span><span>		</span><span>entries</span><span>.</span><span>id</span><span> </span><span>AS</span><span> </span><span>id</span><span>,</span><span>
</span></span></span><span><span><span>		</span><span>json_each</span><span>.</span><span>key</span><span> </span><span>||</span><span> </span><span>&#39; &#39;</span><span> </span><span>||</span><span> </span><span>json_each</span><span>.</span><span>value</span><span> </span><span>AS</span><span> </span><span>kv</span><span>
</span></span></span><span><span><span>	</span><span>FROM</span><span>
</span></span></span><span><span><span>		</span><span>entries</span><span>,</span><span>
</span></span></span><span><span><span>		</span><span>json_each</span><span>(</span><span>entries</span><span>.</span><span>tags</span><span>)</span><span>
</span></span></span><span><span><span></span><span>)</span><span>
</span></span></span><span><span><span></span><span>INSERT</span><span> </span><span>INTO</span><span>
</span></span></span><span><span><span>	</span><span>search</span><span>(</span><span>rowid</span><span>,</span><span> </span><span>tags</span><span>)</span><span>
</span></span></span><span><span><span></span><span>SELECT</span><span>
</span></span></span><span><span><span>	</span><span>id</span><span>,</span><span>
</span></span></span><span><span><span>	</span><span>GROUP_CONCAT</span><span>(</span><span>kv</span><span>,</span><span> </span><span>&#39; &#39;</span><span>)</span><span>
</span></span></span><span><span><span></span><span>FROM</span><span>
</span></span></span><span><span><span>	</span><span>tags</span><span>
</span></span></span><span><span><span></span><span>GROUP</span><span> </span><span>BY</span><span>
</span></span></span><span><span><span>	</span><span>id</span><span>;</span><span>
</span></span></span></code></pre><p>This approach, combined with using a porter tokenization, allowed me to write
fast queries. For example, searching for “Costco” became incredibly fast, under
a millisecond, though it sometimes returned partial matches like “Costco Mart.”</p>
<pre><code><span><span><span>SELECT</span><span> </span><span>rowid</span><span> </span><span>FROM</span><span> </span><span>search</span><span> </span><span>WHERE</span><span> </span><span>search</span><span> </span><span>MATCH</span><span> </span><span>&#34;Costco&#34;</span><span>;</span><span>
</span></span></span></code></pre><p>Queries with tag-specific values (i.e., <code>amenity=cafe</code>) can use text search:</p>
<pre><code><span><span><span>SELECT</span><span> </span><span>rowid</span><span> </span><span>FROM</span><span> </span><span>search</span><span> </span><span>WHERE</span><span> </span><span>search</span><span> </span><span>MATCH</span><span> </span><span>&#34;amenity cafe&#34;</span><span>;</span><span>
</span></span></span></code></pre><p>This will return results with the words <code>amenity</code> and <code>cafe</code> appearing in the
full-text index. It does not ensure that the tag equals that specific value. At
the moment, it is best effort, so there are false positives when returning
results.</p>
<p>Despite these improvements, the 40-gigabyte file size needed to be more
manageable. This is a read-only data set, so there may be ways to compress the
data. There are commercial solutions for this, provided by the core maintainers.
SQLite’s <a href="https://www.sqlite.org/vfs.html" rel="nofollow">virtual file system (VFS)</a> feature
allows an interface for all file operations. This allows different file-like
systems to be used for storage, such as blob stores, other databases, etc.</p>
<p>Initially, I used GZIP compression via Go’s built-in functionality, but it
proved too slow due to the need to decompress large portions of the file for
random reads. It appears that the whole file has to be decompressed before
reading parts of it.</p>
<p>Further research led me to Facebook’s Zstandard (ZSTD) compression, which
supports a
<a href="https://github.com/facebook/zstd/blob/3de0541aef8da51f144ef47fb86dcc38b21afb00/contrib/seekable_format/zstd_seekable_compression_format.md" rel="nofollow">seekable format</a>
suitable for random access reads. This format maps well to SQLite’s
<a href="https://www.sqlite.org/pragma.html#pragma_page_size" rel="nofollow">page size</a> for writing
data to the file.</p>
<p>I could see that compressing the SQLite file with ZSTD reduced its size to about
13 gigabytes. Benchmarking of compressed and uncompressed SQLite databases. This
is a test database of million entries with a random string of text.</p>
<pre><code>BenchmarkReadUncompressedSQLite-4              	  159717	      7459 ns/op	     473 B/op	      15 allocs/op
BenchmarkReadCompressedSQLite-4                	  266703	      3877 ns/op	    2635 B/op	      15 allocs/op
</code></pre>
<p><em>Note:</em> The benchmark (via Go) shows that ZSTD is faster than native. My
hypothesis is that this is because the size of the database can be uncompressed
once and held all in memory.</p>
<p>There was a performance hit with the entire database of OpenStreetMap data. I
believe it has to do with how much data there is compared to the test benchmark
above. However, having a compressed database, where a query cost is still sub-50
milliseconds, is helpful.</p>
<p>I’ve not further optimized the size of the file. I’m pretty happy with this. I
have a <em>TODO</em> for myself to rewrite the ZSTD VFS in C instead of Go.</p>
<p>I want to reduce the number of false positives for the query <code>amenity=cafe</code>.
Using the full-text index, it returns results containing the two words, as tags
are not individually indexed.</p>
<p>When using the FTS5 virtual table, it turns out that constraints can be used on
the original data. The index of the full-text search is used first (according to
the query planner), so we can filter that subset down with a more familiar SQL
constraint.</p>
<pre><code><span><span><span>SELECT</span><span>
</span></span></span><span><span><span>	</span><span>id</span><span>
</span></span></span><span><span><span></span><span>FROM</span><span>
</span></span></span><span><span><span>	</span><span>entries</span><span> </span><span>e</span><span>
</span></span></span><span><span><span>	</span><span>JOIN</span><span> </span><span>search</span><span> </span><span>s</span><span> </span><span>ON</span><span> </span><span>s</span><span>.</span><span>rowid</span><span> </span><span>=</span><span> </span><span>e</span><span>.</span><span>id</span><span>
</span></span></span><span><span><span></span><span>WHERE</span><span>
</span></span></span><span><span><span>	</span><span>-- use FTS index to find subset of possible results
</span></span></span><span><span><span></span><span>	</span><span>search</span><span> </span><span>MATCH</span><span> </span><span>&#39;amenity cafe&#39;</span><span>
</span></span></span><span><span><span>	</span><span>-- use the subset to find exact matches
</span></span></span><span><span><span></span><span>	</span><span>AND</span><span> </span><span>tags</span><span>-&gt;&gt;</span><span>&#39;amenity&#39;</span><span> </span><span>=</span><span> </span><span>&#39;cafe&#39;</span><span>;</span><span>
</span></span></span></code></pre><p>The equals constraint does not use an index, but since it is done on a subset of
results, the operation cost is small. The query is still sub-50ms.</p>
<p>All this provides a read-only SQL queryable data in a single file representing
OpenStreetMap metadata. The project evolved from merely transferring format
migration to optimizing it for efficient search.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2024-07-02.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Deploying a Rails Application with AWS Copilot</title>
      <link>https://jtarchie.com/posts/2023-11-15.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="deploying-a-rails-application-with-aws-copilot"><a href="#deploying-a-rails-application-with-aws-copilot" rel="nofollow">#</a> Deploying a Rails Application with AWS Copilot</h1>
<p>This post outlines deploying a Rails application using AWS Copilot, a CLI tool
designed to provide a Heroku-like experience within the AWS ecosystem. Before
proceeding, it’s recommended to familiarize yourself with the basics by
following the
<a href="https://aws.github.io/copilot-cli/docs/getting-started/first-app-tutorial/" rel="nofollow">“Deploy your first application”</a>
guide in the Copilot documentation.</p>
<p>I used the <a href="https://jumpstartrails.com/" rel="nofollow">Jumpstart Rails</a> project template for
this deployment. The steps and configurations discussed here are specific to
this template but might be adaptable to other Rails applications.</p>
<p><strong>Important Notes:</strong></p>
<ul>
<li>Each <code>copilot</code> command executes CloudFormation operations in the background.
Inspecting the CloudFormation Stacks in the AWS console can be insightful if
you encounter issues.</li>
<li><code>copilot</code> commands can be time-consuming.</li>
</ul>
<h2 id="setting-up-the-application"><a href="#setting-up-the-application" rel="nofollow">#</a> Setting Up the Application</h2>
<p>First, log into the AWS account and set up the application and environment using
the following commands:</p>
<pre><code><span><span><span># Log into the AWS account (using SSO for this deployment)</span>
</span></span><span><span>aws configure sso
</span></span><span><span>
</span></span><span><span><span># Initialize the application repository</span>
</span></span><span><span>copilot init <span>\
</span></span></span><span><span><span></span>  -a rails-app <span>\
</span></span></span><span><span><span></span>  -t <span>&#34;Load Balanced Web Service&#34;</span> <span>\
</span></span></span><span><span><span></span>  -n web <span>\
</span></span></span><span><span><span></span>  -d ./Dockerfile.production
</span></span><span><span>
</span></span><span><span><span># Create a test/development environment</span>
</span></span><span><span>copilot env init <span>\
</span></span></span><span><span><span></span>  --name dev <span>\
</span></span></span><span><span><span></span>  --app rails-app <span>\
</span></span></span><span><span><span></span>  --default-config
</span></span></code></pre><p>This step does not deploy the application but sets up the necessary
configuration within your repository’s <code>copilot/</code> directory.</p>
<h3 id="database-configuration"><a href="#database-configuration" rel="nofollow">#</a> Database Configuration</h3>
<p>Next, allocate the database as a
<a href="https://aws.github.io/copilot-cli/docs/commands/storage-init/" rel="nofollow"><code>storage</code></a> unit:</p>
<pre><code><span><span>copilot storage init <span>\
</span></span></span><span><span><span></span>  -n webdb <span>\
</span></span></span><span><span><span></span>  -t Aurora <span>\
</span></span></span><span><span><span></span>  -w web <span>\
</span></span></span><span><span><span></span>  -l environment <span>\
</span></span></span><span><span><span></span>  --engine PostgreSQL <span>\
</span></span></span><span><span><span></span>  --initial-db railsapp
</span></span></code></pre><p>For the application to access database credentials, modify the <code>manifest.yml</code>
file as follows:</p>
<pre><code><span><span><span>network</span><span>:</span><span>
</span></span></span><span><span><span>  </span><span>vpc</span><span>:</span><span>
</span></span></span><span><span><span>    </span><span>security_groups</span><span>:</span><span>
</span></span></span><span><span><span>      </span>- <span>from_cfn</span><span>:</span><span> </span><span>${COPILOT_APPLICATION_NAME}-${COPILOT_ENVIRONMENT_NAME}-webdbSecurityGroup</span><span>
</span></span></span><span><span><span></span><span>secrets</span><span>:</span><span>
</span></span></span><span><span><span>  </span><span>DATABASE_JSON</span><span>:</span><span>
</span></span></span><span><span><span>    </span><span>from_cfn</span><span>:</span><span> </span><span>${COPILOT_APPLICATION_NAME}-${COPILOT_ENVIRONMENT_NAME}-webdbAuroraSecret</span><span>
</span></span></span></code></pre><h3 id="application-configuration"><a href="#application-configuration" rel="nofollow">#</a> Application Configuration</h3>
<p>AWS provides <code>DATABASE_JSON</code> instead of the typical <code>DATABASE_URL</code>. Adjust the
<code>database.yml</code> to parse these credentials:</p>
<pre><code><span><span><span>production</span><span>:</span><span>
</span></span></span><span><span><span>  </span><span>&lt;&lt;</span><span>:</span><span> </span><span>*default</span><span>
</span></span></span><span><span><span>  </span><span># Parse DATABASE_JSON if available</span><span>
</span></span></span><span><span><span>  </span><span>&lt;% if ENV[&#39;DATABASE_JSON&#39;] %&gt;</span><span>
</span></span></span><span><span><span>  </span><span>&lt;%   require &#39;json&#39; %&gt;</span><span>
</span></span></span><span><span><span>  </span><span>&lt;%   db_config = JSON.parse(ENV[&#39;DATABASE_JSON&#39;]) %&gt;</span><span>
</span></span></span><span><span><span>  </span><span>username</span><span>:</span><span> </span><span>&lt;%= db_config[&#39;username&#39;] %&gt;</span><span>
</span></span></span><span><span><span>  </span><span>password</span><span>:</span><span> </span><span>&lt;%= db_config[&#39;password&#39;] %&gt;</span><span>
</span></span></span><span><span><span>  </span><span>host</span><span>:</span><span> </span><span>&lt;%= db_config[&#39;host&#39;] %&gt;</span><span>
</span></span></span><span><span><span>  </span><span>port</span><span>:</span><span> </span><span>&lt;%= db_config[&#39;port&#39;] %&gt;</span><span>
</span></span></span><span><span><span>  </span><span>database</span><span>:</span><span> </span><span>&lt;%= db_config[&#39;dbname&#39;] %&gt;</span><span>
</span></span></span><span><span><span>  </span><span>&lt;% else %&gt;</span><span>
</span></span></span><span><span><span>  </span><span>url</span><span>:</span><span> </span><span>&lt;%= ENV[&#39;DATABASE_URL&#39;] %&gt;</span><span>
</span></span></span><span><span><span>  </span><span>&lt;% end %&gt;</span><span>
</span></span></span></code></pre><p>In your <code>Dockerfile.production</code>, include instructions for precompiling assets
and starting the Rails server:</p>
<pre><code><span><span><span>RUN</span> <span>SECRET_KEY_BASE</span><span>=</span>dummy-staging-key bin/rails assets:precompile<span>
</span></span></span><span><span><span></span><span>CMD</span> <span>[</span><span>&#34;bin/rails&#34;</span><span>,</span> <span>&#34;server&#34;</span><span>,</span> <span>&#34;-b&#34;</span><span>,</span> <span>&#34;0.0.0.0&#34;</span><span>]</span><span>
</span></span></span></code></pre><p>Set environment variables for production in your <code>manifest.yml</code>:</p>
<pre><code><span><span><span>variables</span><span>:</span><span>
</span></span></span><span><span><span>  </span><span>RAILS_ENV</span><span>:</span><span> </span><span>production</span><span>
</span></span></span><span><span><span>  </span><span>SECRET_KEY_BASE</span><span>:</span><span> </span><span>&lt;value from `rails secret`&gt;</span><span>
</span></span></span></code></pre><h3 id="service-configuration"><a href="#service-configuration" rel="nofollow">#</a> Service Configuration</h3>
<p>Configure the application as a
<a href="https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/" rel="nofollow">“Load Balanced Web Service”</a>
to enable command execution in the container, essential for tasks like database
migration.</p>
<pre><code><span><span><span>exec</span><span>:</span><span> </span><span>true</span><span>
</span></span></span></code></pre><p>Define routing and health check settings:</p>
<pre><code><span><span><span>http</span><span>:</span><span>
</span></span></span><span><span><span>  </span><span>path</span><span>:</span><span> </span><span>&#34;/&#34;</span><span>
</span></span></span><span><span><span>  </span><span>healthcheck</span><span>:</span><span> </span><span>&#34;/up&#34;</span><span>
</span></span></span></code></pre><p>Since SSL support isn’t provided by default, turn off SSL enforcement in
<code>config/environments/production.rb</code>:</p>
<pre><code><span><span><span>config</span><span>.</span><span>force_ssl</span> <span>=</span> <span>false</span>
</span></span></code></pre><h3 id="deploying-the-application"><a href="#deploying-the-application" rel="nofollow">#</a> Deploying the Application</h3>
<p>Deploy the application and database using the following commands:</p>
<pre><code><span><span>copilot deploy <span>\
</span></span></span><span><span><span></span>  --deploy-env <span>\
</span></span></span><span><span><span></span>  --env dev <span>\
</span></span></span><span><span><span></span>  --name web
</span></span><span><span>
</span></span><span><span><span># Run database migrations post-deployment</span>
</span></span><span><span>copilot svc <span>exec</span> <span>\
</span></span></span><span><span><span></span>  --command <span>&#34;bin/rails db:migrate&#34;</span> <span>\
</span></span></span><span><span><span></span>  -n web
</span></span><span><span>
</span></span><span><span><span># View application logs</span>
</span></span><span><span>copilot svc logs
</span></span><span><span>
</span></span><span><span><span># Display deployment information</span>
</span></span><span><span>copilot svc show
</span></span></code></pre><p>You should have a URL in the <code>show</code> command that will expose your app to the
public Internet.</p>
<p>We did it! We’ve deployed our application.</p>
<p>This has been the best application tool to deploy into the AWS ecosystem. This
is, however, the longest time it has taken me to deploy a hello world Rails
application with compromises – i.e., SSL, one of the tasks, etc.</p>
<p>How can I help make this experience better for others after me? I have a blog
post to write about the incident. I do want the experience to be more flawless.
There are a lot of gothyas and “didn’t you reads” that have been discoverable
but disappointing.</p>
<blockquote>
<p>Note: To clean it all up <code>copilot app delete</code>.</p>
</blockquote>
<h3 id="additional-resources"><a href="#additional-resources" rel="nofollow">#</a> Additional Resources</h3>
<ul>
<li><a href="https://www.endpointdev.com/blog/2022/06/how-to-deploy-django-app-with-aurora-serverless-and-copilot/" rel="nofollow">Deploy Django App</a>:
Offers insights into web app deployment in the cloud, including database
configuration.</li>
<li><a href="https://github.com/aws/copilot-cli/issues/2071" rel="nofollow">SSL Support Discussion</a></li>
</ul>
<h2 id="update"><a href="#update" rel="nofollow">#</a> Update</h2>
<p>I was later informed via a
<a href="https://matrix.to/#/!QdxoBcgpJveoAoIPCc:gitter.im/$YqOcVQ2VqEqsXvDqEYeOsEOR-mp7HhDbLoBJCw67J8I?via=gitter.im&amp;via=matrix.org&amp;via=matrix.unope.ru" rel="nofollow">chat thread</a>
with the team, that I can get SSL using <code>Request-Driven Web Service</code>.</p>
<pre><code><span><span>copilot init <span>\
</span></span></span><span><span><span></span>  -a rails-app <span>\
</span></span></span><span><span><span></span>  -t <span>&#34;Request-Driven Web Service&#34;</span> <span>\
</span></span></span><span><span><span></span>  -n web <span>\
</span></span></span><span><span><span></span>  -d ./Dockerfile.production
</span></span></code></pre><blockquote>
<p>Note: When setting up the database the instructions are bit different, but the
cli will tell you what to add.</p>
</blockquote>
<p>The issue that I was running into of not being able database migrations can be
solved with
<a href="https://aws.github.io/copilot-cli/docs/commands/task-run/" rel="nofollow"><code>copilot task run</code></a>.</p>
<pre><code><span><span>copilot task run <span>\
</span></span></span><span><span><span></span>  --command <span>&#34;bin/rails db:migrate&#34;</span> <span>\
</span></span></span><span><span><span></span>  --dockerfile Dockerfile.production <span>\
</span></span></span><span><span><span></span>  --env dev <span>\
</span></span></span><span><span><span></span>  --app rails-app <span>\
</span></span></span><span><span><span></span>  --follow
</span></span></code></pre><p>However, I found a bug with <code>task run</code>. When it builds the docker image, it uses
the host architecture, it is not building for the target architecture. I changed
my command to use the latest image of the app I just deployed, using
<code>docker images</code>.</p>
<pre><code><span><span>copilot task run <span>\
</span></span></span><span><span><span></span>  --command <span>&#34;bin/rails db:migrate&#34;</span> <span>\
</span></span></span><span><span><span></span>  --image &lt;ecr host&gt;/rails-app/web <span>\
</span></span></span><span><span><span></span>  --env dev <span>\
</span></span></span><span><span><span></span>  --app rails-app <span>\
</span></span></span><span><span><span></span>  --follow
</span></span></code></pre><p>This did not work because the <code>task run</code> does not assume the environment
variables and secrets from the services. This would include <code>RAILS_ENV</code> and the
<code>DATABASE_JSON</code>.</p>
<p>This does not appear to be a viable way for me.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-11-15.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Implementing a simple Mutex in Go</title>
      <link>https://jtarchie.com/posts/2023-10-30.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="implementing-a-simple-mutex-in-go"><a href="#implementing-a-simple-mutex-in-go" rel="nofollow">#</a> Implementing a simple Mutex in Go</h1>
<p>When asked, “How do you learn concurrency in Go?” my default answer is to create
a web application. But Go has made building web applications so straightforward
that I’ve been curious about diving deeper into Go’s concurrency primitives.</p>
<p>So, why not construct a Mutex from scratch? While Go’s standard library already
provides one, it would be helpful to understand why. Note that this won’t be an
exhaustive tutorial on Go’s concurrency but more of a hands-on exploration.</p>
<blockquote>
<p>Note: This is an introduction to the concept and implementation. It will not
end up being a production-quality mutex.</p>
</blockquote>
<p>A mutex, at its core, is a lock that ensures exclusive access to the execution
of code. This helps prevent synchronization errors, ensures data integrity, and
limits bad actors.</p>
<p>Consider the banking dilemma. If two transactions—a withdrawal and a
deposit—coincide, they can interfere because of the amount of the account
balance. If we start with an account balance of $100, have a withdrawal ($20)
and deposit ($30) occur, we should have $110 at the end of all transactions.</p>
<p>Here’s a flow chart to illustrate the potential problem:</p>
<pre>%%{init: {&#39;theme&#39;:&#39;forest&#39;}}%%
sequenceDiagram
    participant Account as Balance
    participant Withdrawal as Withdrawal Process
    participant Deposit as Deposit Process

    Note over Account: Starting balance:&lt;br&gt;$100

    Withdrawal-&gt;&gt;Account: Request to withdraw $20
    Deposit-&gt;&gt;Account: Request to deposit $30
    Note over Withdrawal,Deposit: These operations occur nearly&lt;br&gt;at the same time.

    Account--&gt;&gt;Withdrawal: Check balance (Sees $100)
    Account--&gt;&gt;Deposit: Check balance (Sees $100)

    Account--&gt;&gt;Withdrawal: Authorizes withdrawal.
    Account--&gt;&gt;Deposit: Authorizes deposit.

    Note over Account: Expected balance after&lt;br&gt;both operations: $110.

    Withdrawal-&gt;&gt;Account: Deducts $20. New balance: $80
    Deposit-&gt;&gt;Account: Adds $30. New balance: $130.

    Note over Account: Oops! Final balance is $130 or $80&lt;br&gt;instead of the expected $110.
</pre><p>By introducing a mutex or a lock, we can control access, ensuring only one
transaction can read or update the account balance at any given time. It’s as
simple as that!</p>
<p>For a hands-on approach, we’ll use test-driven development. Our test will
simulate the scenario above. For this, we’ll leverage goroutines in Go, enabling
us to run code concurrently.</p>
<pre><code><span><span><span>package</span> <span>main_test</span>
</span></span><span><span>
</span></span><span><span><span>import</span> <span>(</span>
</span></span><span><span>  <span>&#34;testing&#34;</span>
</span></span><span><span>  <span>&#34;time&#34;</span>
</span></span><span><span><span>)</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>TestRaceCondition</span><span>(</span><span>t</span> <span>*</span><span>testing</span><span>.</span><span>T</span><span>)</span> <span>{</span>
</span></span><span><span>  <span>balance</span> <span>:=</span> <span>100</span>
</span></span><span><span>
</span></span><span><span>  <span>// Concurrent withdrawal
</span></span></span><span><span><span></span>  <span>go</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>    <span>balance</span> <span>=</span> <span>balance</span> <span>-</span> <span>20</span>
</span></span><span><span>  <span>}()</span>
</span></span><span><span>
</span></span><span><span>  <span>// Concurrent deposit
</span></span></span><span><span><span></span>  <span>go</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>    <span>balance</span> <span>=</span> <span>balance</span> <span>+</span> <span>30</span>
</span></span><span><span>  <span>}()</span>
</span></span><span><span>
</span></span><span><span>  <span>if</span> <span>balance</span> <span>!=</span> <span>110</span> <span>{</span>
</span></span><span><span>    <span>t</span><span>.</span><span>Errorf</span><span>(</span><span>&#34;Expected balance to be $100, but got $%d&#34;</span><span>,</span> <span>balance</span><span>)</span>
</span></span><span><span>  <span>}</span>
</span></span><span><span><span>}</span>
</span></span></code></pre><p>Executing <code>go test main_test.go</code> might pass successfully on the first try. But
does that mean we’re in the clear? Not quite. When you run the code with the
race detector, <code>go test -race main_test.go</code>, it will flag a <code>DATA RACE</code>.</p>
<pre><code>==================
WARNING: DATA RACE
Write at 0x00c000124158 by goroutine 8:
  command-line-arguments_test.TestRaceCondition.func2()
      /main_test.go:13 +0x3c

Previous read at 0x00c000124158 by goroutine 6:
  command-line-arguments_test.TestRaceCondition()
      /main_test.go:16 +0x110
</code></pre>
<p>This warning indicates simultaneous read and write operations on the <code>balance</code>
variable. The <code>-race</code> flag is an invaluable feature in Go’s toolkit, designed to
highlight such concurrent data access issues.</p>
<p>We must have each asynchronous operation access <code>balance</code> one at a time. The
approach will be to have the process ask for permission, either get or wait for
access, perform the operation, and release access. This will be the logic for
our mutex.</p>
<p>The baseline for our mutex will be if access to the data is requested or not.</p>
<p>To follow the logic above, our code would look like the following:</p>
<pre><code><span><span>  <span>accessible</span> <span>:=</span> <span>false</span>
</span></span><span><span>
</span></span><span><span>	<span>lock</span> <span>:=</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>		<span>for</span> <span>{</span>
</span></span><span><span>			<span>if</span> <span>!</span><span>accessible</span> <span>{</span>
</span></span><span><span>				<span>accessible</span> <span>=</span> <span>true</span>
</span></span><span><span>				<span>return</span>
</span></span><span><span>			<span>}</span>
</span></span><span><span>		<span>}</span>
</span></span><span><span>	<span>}</span>
</span></span><span><span>
</span></span><span><span>	<span>unlock</span> <span>:=</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>		<span>accessible</span> <span>=</span> <span>false</span>
</span></span><span><span>	<span>}</span>
</span></span></code></pre><p>We can then add <code>lock</code> and <code>unlock</code> in the areas we access <code>balance</code>.</p>
<pre><code><span><span>  <span>go</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>    <span>lock</span><span>()</span>
</span></span><span><span>    <span>defer</span> <span>unlock</span><span>()</span>
</span></span><span><span>
</span></span><span><span>    <span>balance</span> <span>=</span> <span>balance</span> <span>-</span> <span>20</span>
</span></span><span><span>  <span>}()</span>
</span></span><span><span>
</span></span><span><span>  <span>go</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>    <span>lock</span><span>()</span>
</span></span><span><span>    <span>defer</span> <span>unlock</span><span>()</span>
</span></span><span><span>
</span></span><span><span>    <span>balance</span> <span>=</span> <span>balance</span> <span>+</span> <span>30</span>
</span></span><span><span>  <span>}()</span>
</span></span><span><span>
</span></span><span><span>  <span>lock</span><span>()</span>
</span></span><span><span>	<span>defer</span> <span>unlock</span><span>()</span>
</span></span><span><span>
</span></span><span><span>	<span>if</span> <span>balance</span> <span>!=</span> <span>110</span> <span>{</span>
</span></span><span><span>		<span>t</span><span>.</span><span>Errorf</span><span>(</span><span>&#34;Expected balance to be $110, but got $%d&#34;</span><span>,</span> <span>balance</span><span>)</span>
</span></span><span><span>	<span>}</span>
</span></span></code></pre><p>This implementation, however, does not work. It depends on a single value, which
itself causes a race condition. With <code>lock,</code> we are reading the value and then
modifying it. With more than one for loop, nothing guarantees that <code>accessible</code>
isn’t also be accessed at the same time, like <code>balance.</code></p>
<p>The Golang standard library is a way of ensuring a value can only be accessed
once at a time across goroutines. The
<a href="https://pkg.go.dev/sync/atomic" rel="nofollow"><code>atomic</code> package</a> includes functions to load
and store values in a thread-safe runtime.</p>
<blockquote>
<p>Note: We can use the <code>atomic</code> with <code>balance,</code> but let’s implement a mutex
first; that’s what we are building.</p>
</blockquote>
<p>Let’s rewrite <code>lock</code> and <code>unlock</code> to use the <code>atomic</code> functions. With the
standard</p>
<pre><code><span><span>  <span>var</span> <span>accessible</span> <span>int32</span>
</span></span><span><span>
</span></span><span><span>	<span>lock</span> <span>:=</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>		<span>for</span> <span>{</span>
</span></span><span><span>			<span>if</span> <span>atomic</span><span>.</span><span>LoadInt32</span><span>(</span><span>&amp;</span><span>accessible</span><span>)</span> <span>==</span> <span>0</span> <span>{</span>
</span></span><span><span>				<span>atomic</span><span>.</span><span>StoreInt32</span><span>(</span><span>&amp;</span><span>accessible</span><span>,</span> <span>1</span><span>)</span>
</span></span><span><span>				<span>return</span>
</span></span><span><span>			<span>}</span>
</span></span><span><span>		<span>}</span>
</span></span><span><span>	<span>}</span>
</span></span><span><span>
</span></span><span><span>	<span>unlock</span> <span>:=</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>		<span>atomic</span><span>.</span><span>StoreInt32</span><span>(</span><span>&amp;</span><span>accessible</span><span>,</span> <span>0</span><span>)</span>
</span></span><span><span>	<span>}</span>
</span></span></code></pre><p>When running our tests now, we can see that they pass – sometimes. A data race
happens every few runs. The trace shows that accessing the <code>balance</code> value for
our final assertion is an issue. This information indicates that our mutex is
working by definition. The race condition no longer occurs within the
goroutines.</p>
<p>To fix our assertion so it is thread-safe, we must ensure that the goroutines
have time to run and then use the mutex to access <code>balance.</code> Let’s do the
cheapest way possible, a <code>time.Sleep</code> for the go runtime.</p>
<pre><code><span><span>	<span>time</span><span>.</span><span>Sleep</span><span>(</span><span>time</span><span>.</span><span>Millisecond</span><span>)</span>
</span></span><span><span>
</span></span><span><span>	<span>lock</span><span>()</span>
</span></span><span><span>	<span>defer</span> <span>unlock</span><span>()</span>
</span></span><span><span>
</span></span><span><span>	<span>if</span> <span>balance</span> <span>!=</span> <span>110</span> <span>{</span>
</span></span><span><span>		<span>t</span><span>.</span><span>Errorf</span><span>(</span><span>&#34;Expected balance to be $110, but got $%d&#34;</span><span>,</span> <span>balance</span><span>)</span>
</span></span><span><span>	<span>}</span>
</span></span></code></pre><p>Running the tests (with the race condition) works correctly and consistently. We
created a mutex! Is it done right?</p>
<p>We can refactor our code. Above, I purposely introduced functionality from
<code>atomic</code> for loading and storing so we could implement using our original mutex
logic.</p>
<p>The <code>atomic</code> library (since go 1.19) can handle boolean values. The
<a href="https://pkg.go.dev/sync/atomic#Bool" rel="nofollow"><code>Bool</code></a> type supports the operation to set
a value only if an expected value exists – via <code>CompareAndSwap</code>.</p>
<p>Our implementation changes to the following:</p>
<pre><code><span><span>  <span>accessible</span> <span>:=</span> <span>&amp;</span><span>atomic</span><span>.</span><span>Bool</span><span>{}</span>
</span></span><span><span>
</span></span><span><span>	<span>lock</span> <span>:=</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>		<span>for</span> <span>{</span>
</span></span><span><span>			<span>if</span> <span>accessible</span><span>.</span><span>CompareAndSwap</span><span>(</span><span>false</span><span>,</span> <span>true</span><span>)</span> <span>{</span>
</span></span><span><span>				<span>return</span>
</span></span><span><span>			<span>}</span>
</span></span><span><span>		<span>}</span>
</span></span><span><span>	<span>}</span>
</span></span><span><span>
</span></span><span><span>	<span>unlock</span> <span>:=</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>		<span>accessible</span><span>.</span><span>Store</span><span>(</span><span>false</span><span>)</span>
</span></span><span><span>	<span>}</span>
</span></span></code></pre><p>The logic is <em>simple</em> using the functionality from <code>atomic</code>. This can eat CPU
cycles while waiting for the unlocking of another segment of code. The go
runtime appears to
<a href="https://github.com/golang/go/issues/10958" rel="nofollow">preemptive scheduling</a> to prevent
this tight loop. We want to do our best not to block other goroutines from
running.</p>
<p>When looking at the Go implementation of
<a href="https://pkg.go.dev/sync#Mutex.Lock" rel="nofollow"><code>sync.Mutex</code></a>, the implementation is
similar for a fast path. There is extensive implementation to prevent CPU
starvation with the <code>for</code> loop in a goroutine. There’s more we could do to build
out our mutex implementation, but this works for our accounting example. A more
extensive example and implementation is another exercise (maybe a follow-up
post).</p>
<p>However, as a final exercise, we don’t need a mutex for the value for <code>balance.</code>
As noted above, the <code>atomic</code> library allows us to load, store, and swap with an
integer value. We can avoid the usage of a mutex altogether.</p>
<pre><code><span><span>	<span>balance</span> <span>:=</span> <span>&amp;</span><span>atomic</span><span>.</span><span>Int32</span><span>{}</span>
</span></span><span><span>	<span>balance</span><span>.</span><span>Store</span><span>(</span><span>100</span><span>)</span>
</span></span><span><span>
</span></span><span><span>	<span>go</span> <span>balance</span><span>.</span><span>Add</span><span>(</span><span>-</span><span>20</span><span>)</span>
</span></span><span><span>	<span>go</span> <span>balance</span><span>.</span><span>Add</span><span>(</span><span>30</span><span>)</span>
</span></span><span><span>
</span></span><span><span>	<span>time</span><span>.</span><span>Sleep</span><span>(</span><span>time</span><span>.</span><span>Millisecond</span><span>)</span>
</span></span><span><span>
</span></span><span><span>	<span>if</span> <span>balance</span><span>.</span><span>Load</span><span>()</span> <span>!=</span> <span>110</span> <span>{</span>
</span></span><span><span>		<span>t</span><span>.</span><span>Errorf</span><span>(</span><span>&#34;Expected balance to be $110, but got $%d&#34;</span><span>,</span> <span>balance</span><span>)</span>
</span></span><span><span>	<span>}</span>
</span></span></code></pre><p>Thanks for following along. Discussion on
<a href="https://www.reddit.com/r/golang/comments/17n0djm/implementing_a_simple_mutex/" rel="nofollow">Reddit</a>.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-10-30.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Transitioning Back into the Labor Force</title>
      <link>https://jtarchie.com/posts/2023-10-14.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="transitioning-back-into-the-labor-force"><a href="#transitioning-back-into-the-labor-force" rel="nofollow">#</a> Transitioning Back into the Labor Force</h1>
<aside>This is a continuation of <a href="2023-10-08.html" rel="nofollow">Taking Time Off Between Jobs</a></aside>
<p>I received many responses after sharing my experiences about taking time off
between jobs. Many resonated with my sentiments, while others sought deeper
insights into transitioning back into the workforce. Today, I delve into these
inquiries and illuminate my journey back into the labor force.</p>
<p>Firstly, I want to acknowledge a crucial point: not everyone can take time off.
Financial, familial, and health commitments can often tether us to our roles. I
recognize the privilege I hold as a software engineer, a profession that has
granted me diverse employment opportunities and financial freedom.</p>
<p>A recurring question I’ve encountered centers around the challenges of reentry.
Specifically, how does one make their return appealing to potential employers
after a significant hiatus?</p>
<p>The bias against career gaps is undeniable. While some perceive it as “time off
from work,” I’ve always viewed it as “time taken for myself.” Thankfully, many
of my employers have been understanding and empathetic. Their interview
processes, likely influenced by the nature of software engineering, prioritize
technical expertise over employment gaps.</p>
<p>During my breaks, coding remained a constant passion. My GitHub profile is a
testament to this, bustling with activity as I explore application ideas, some
whimsical and others with potential. Leaving some projects incomplete is
perfectly acceptable; it’s all part of the journey.</p>
<p>However, only some interactions have been positive. I distinctly remember a
recruiter who condescendingly, commented on my “little timey offy.” While such
biases are disheartening, I’ve realized that it’s impossible to shift every
perspective, especially within the constraints of an interview. Instead, I’ve
become adept at explaining my breaks, highlighting the flexibility they offer in
interviewing without needing to take PTO.</p>
<p>On the rare occasions when my reasons were probed further, I emphasized my
intent to find the proper role. In more intrusive situations, I’ve gently
reminded interviewers that some might take breaks for the reasons others can’t,
be it health, family, or financial concerns. It’s crucial to maintain personal
boundaries. Just as one wouldn’t necessarily answer a police officer’s question
about perceived wrongdoing, it’s best to keep job interviews focused on relevant
topics.</p>
<p>Reentering the workforce has been relatively seamless, though finding the right
reentry point can be time-consuming. When job hunting, I prioritize strong
culture, effective communication, and the ability to prioritize, which can
extend the search.</p>
<p>While my perspective is specific to software engineering, I’m curious if others
in different professions have had similar experiences.</p>
<p>I advocate for normalizing career breaks. Everyone has their reasons, and we
must respect and understand them. Whether contemplating a break or rejoining the
workforce, remember that your journey is uniquely yours, and there’s no singular
correct path.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-10-14.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Taking Time Off Between Jobs</title>
      <link>https://jtarchie.com/posts/2023-10-08.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="taking-time-off-between-jobs"><a href="#taking-time-off-between-jobs" rel="nofollow">#</a> Taking Time Off Between Jobs</h1>
<aside>A follow up was written <a href="2023-10-14.html" rel="nofollow">Transitioning Back into the Labor Force</a></aside>
<p>Amid our fast-paced lives, taking time off between jobs might seem daunting. But
as I reflect on my journey, I want to share the profound impact these pauses
have had on my life and why you might consider embracing them, too.</p>
<p>Years ago, my father jokingly suggested I write about how to take time off
between jobs. Today, as I see friends and co-workers navigating career
transitions, I realize the importance of discussing this topic. It’s a path I’ve
walked out of sheer burnout and necessity.</p>
<p>My first experience with taking time off was after a grueling stint at a
startup. I clocked in over 80 hours weekly, often sleeping in the office to meet
deadlines. The stress was palpable. Despite the potential skepticism of former
colleagues, I felt like I was always the first one in and the last one out. One
day, I decided I couldn’t continue. I left, and the following week, I was
bedridden, my body finally succumbing to the exhaustion.</p>
<p>This period of rest wasn’t just about physical recovery. It was a time of
introspection. I had to relearn how to relax to understand that not every day
needed a milestone. This realization was transformative. I began to explore,
interviewing for various roles and eventually landing a job at Pivotal Labs in
San Francisco. With its myriad tech opportunities, this city was a playground
for personal and professional growth.</p>
<p>Years later, I found myself in New York City after another job transition. With
the lessons from my previous break, I explored the city with vigor, visiting
coffee shops, exploring boroughs, and even traveling to Berlin. The freedom to
interview for jobs without the constraints of a 9-5 was liberating.</p>
<p>However, life is unpredictable. After a short stint at another company, I faced
layoffs. By 2023, I decided to take another break. This time, the world was
different. The pandemic has reshaped our lives, our relationships, and our
values. This break allowed me to reconnect with family, help friends, and, most
importantly, reflect on my journey.</p>
<p>Now, as I interview for new roles, I do so with clarity about what I want and
where I want to be. The opportunity to take time off is about more than
relaxation. It’s about growth. It’s about living life beyond the confines of a
9-5 routine. The pandemic has shown us that flexibility is possible, and we
should seize it.</p>
<p>So, what’s my advice to you? Live. Relax. Learn. Explore. Whether picking up a
new hobby or simply taking a moment to breathe, remember that this time off is
<em>your</em> time. The wording misrepresents itself. It’s not about being idle; it’s
about growth, reflection, and understanding oneself better.</p>
<p>Remember that there’s more to life in a world where our jobs often define us.
Don’t be your harshest critic. Embrace the pauses, for they might lead you to
the most fulfilling chapters of your life.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-10-08.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>My Preferred Go Stack</title>
      <link>https://jtarchie.com/posts/2023-09-30.html</link>
      <description>Explore my preferred collection of Go libraries that ergonomically enhance my coding style and streamline the development of Go applications, allowing for efficient and organized coding.&#xA;</description>
      <content:encoded><![CDATA[<h1 id="my-preferred-go-stack"><a href="#my-preferred-go-stack" rel="nofollow">#</a> My Preferred Go Stack</h1>
<p>A few weeks ago, a friend asked how I wrote code so swiftly. The secret isn’t
maintaining a template repo but having a set of preferred libraries that
harmonize with my coding style. In this post, I will introduce you to these
libraries, sharing insights on their benefits and providing example code
snippets to illustrate their ease of use. I encourage you to explore the
extended documentation linked to understand each library’s full scope and
capabilities.</p>
<ul>
<li>
<p><a href="https://github.com/alecthomas/kong" rel="nofollow"><code>kong</code></a></p>
<p>Kong allows me to parse arguments into a struct, providing features like
passing state to the executing command and supporting various arguments. Its
command patterns facilitate the use of global options passed to subcommands.</p>
<pre><code><span><span><span>package</span> <span>main</span>
</span></span><span><span>
</span></span><span><span><span>type</span> <span>CLI</span> <span>struct</span> <span>{</span>
</span></span><span><span>  <span>Filename</span> <span>*</span><span>os</span><span>.</span><span>File</span> <span>`help:&#34;the filename to write&#34;`</span>
</span></span><span><span>  <span>Write</span>    <span>bool</span>     <span>`help:&#34;write something to the file&#34;`</span>
</span></span><span><span><span>}</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>(</span><span>c</span> <span>*</span><span>CLI</span><span>)</span> <span>Run</span><span>()</span> <span>error</span> <span>{</span>
</span></span><span><span>  <span>if</span> <span>c</span><span>.</span><span>Write</span> <span>{</span>
</span></span><span><span>    <span>_</span><span>,</span> <span>err</span> <span>:=</span> <span>c</span><span>.</span><span>Filename</span><span>.</span><span>Write</span><span>([]</span><span>byte</span><span>(</span><span>&#34;Hello World&#34;</span><span>))</span>
</span></span><span><span>    <span>return</span> <span>err</span>
</span></span><span><span>  <span>}</span>
</span></span><span><span>  <span>return</span> <span>nil</span>
</span></span><span><span><span>}</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>main</span><span>()</span> <span>{</span>
</span></span><span><span>  <span>cli</span> <span>:=</span> <span>&amp;</span><span>builder</span><span>.</span><span>CLI</span><span>{}</span>
</span></span><span><span>  <span>ctx</span> <span>:=</span> <span>kong</span><span>.</span><span>Parse</span><span>(</span><span>cli</span><span>)</span>
</span></span><span><span>  <span>err</span> <span>:=</span> <span>ctx</span><span>.</span><span>Run</span><span>()</span>
</span></span><span><span>  <span>ctx</span><span>.</span><span>FatalIfErrorf</span><span>(</span><span>err</span><span>)</span>
</span></span><span><span><span>}</span>
</span></span></code></pre></li>
<li>
<p><a href="https://github.com/imroc/req" rel="nofollow"><code>req</code></a></p>
<p>While Go does offer a HTTP client in its standard library, <code>req</code> eliminates
the boilerplate that comes with it. It features middleware extension, a retry
mechanism, and excellent defaults, such as timeouts, providing transparency in
HTTP requests through its dead mode.</p>
<pre><code><span><span><span>package</span> <span>main</span>
</span></span><span><span>
</span></span><span><span><span>import</span> <span>(</span>
</span></span><span><span>  <span>&#34;fmt&#34;</span>
</span></span><span><span>  <span>&#34;github.com/imroc/req/v3&#34;</span>
</span></span><span><span><span>)</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>main</span><span>()</span> <span>{</span>
</span></span><span><span>  <span>client</span> <span>:=</span> <span>req</span><span>.</span><span>C</span><span>()</span>
</span></span><span><span>  <span>resp</span><span>,</span> <span>err</span> <span>:=</span> <span>client</span><span>.</span><span>R</span><span>().</span>
</span></span><span><span>    <span>SetRetryCount</span><span>(</span><span>3</span><span>).</span>
</span></span><span><span>    <span>Get</span><span>(</span><span>&#34;https://httpbin.org/uuid&#34;</span><span>)</span>
</span></span><span><span>  <span>if</span> <span>err</span> <span>!=</span> <span>nil</span> <span>{</span>
</span></span><span><span>    <span>log</span><span>.</span><span>Fatal</span><span>(</span><span>err</span><span>)</span>
</span></span><span><span>  <span>}</span>
</span></span><span><span>  <span>fmt</span><span>.</span><span>Println</span><span>(</span><span>resp</span><span>)</span>
</span></span><span><span><span>}</span>
</span></span></code></pre></li>
<li>
<p><a href="https://github.com/labstack/echo" rel="nofollow"><code>echo</code></a></p>
<p>Its ability to handle routing and extend middleware, is invaluable for those
coming from a full-stack background. It resembles Sinatra in Ruby and provides
a meticulously documented, excellent routing system.</p>
<pre><code><span><span><span>package</span> <span>main</span>
</span></span><span><span>
</span></span><span><span><span>import</span> <span>(</span>
</span></span><span><span>  <span>&#34;github.com/labstack/echo/v4&#34;</span>
</span></span><span><span><span>)</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>main</span><span>()</span> <span>{</span>
</span></span><span><span>  <span>e</span> <span>:=</span> <span>echo</span><span>.</span><span>New</span><span>()</span>
</span></span><span><span>  <span>e</span><span>.</span><span>GET</span><span>(</span><span>&#34;/hello&#34;</span><span>,</span> <span>hello</span><span>)</span>
</span></span><span><span>  <span>e</span><span>.</span><span>Logger</span><span>.</span><span>Fatal</span><span>(</span><span>e</span><span>.</span><span>Start</span><span>(</span><span>&#34;:1323&#34;</span><span>))</span>
</span></span><span><span><span>}</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>hello</span><span>(</span><span>c</span> <span>echo</span><span>.</span><span>Context</span><span>)</span> <span>error</span> <span>{</span>
</span></span><span><span>  <span>return</span> <span>c</span><span>.</span><span>String</span><span>(</span><span>http</span><span>.</span><span>StatusOK</span><span>,</span> <span>&#34;Hello, World!&#34;</span><span>)</span>
</span></span><span><span><span>}</span>
</span></span></code></pre></li>
<li>
<p><a href="https://github.com/jtarchie/worker" rel="nofollow"><code>worker</code></a></p>
<p>This is my own creation to manage go routine pools and streamline parallel
work, reducing the repetitive coding in starting a routine and having a sync
wait group. It offers configurable go routines and a buffer channel to queue
the work onto.</p>
<pre><code><span><span><span>package</span> <span>main</span>
</span></span><span><span>
</span></span><span><span><span>import</span> <span>(</span>
</span></span><span><span>  <span>&#34;github.com/jtarchie/worker&#34;</span>
</span></span><span><span><span>)</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>main</span><span>()</span> <span>{</span>
</span></span><span><span>  <span>pool</span> <span>:=</span> <span>worker</span><span>.</span><span>New</span><span>[</span><span>int</span><span>](</span><span>10</span><span>,</span> <span>10</span><span>,</span> <span>func</span><span>(</span><span>index</span><span>,</span> <span>value</span> <span>int</span><span>)</span> <span>{</span>
</span></span><span><span>    <span>// Processing in the background here with the value
</span></span></span><span><span><span></span>  <span>})</span>
</span></span><span><span>  <span>defer</span> <span>pool</span><span>.</span><span>Close</span><span>()</span>
</span></span><span><span>
</span></span><span><span>  <span>for</span> <span>i</span> <span>:=</span> <span>0</span><span>;</span> <span>i</span> <span>&lt;</span> <span>100_000</span><span>;</span> <span>i</span><span>++</span> <span>{</span>
</span></span><span><span>    <span>w</span><span>.</span><span>Enqueue</span><span>(</span><span>i</span><span>)</span>
</span></span><span><span>  <span>}</span>
</span></span><span><span><span>}</span>
</span></span></code></pre></li>
<li>
<p><a href="https://github.com/mattn/go-sqlite3" rel="nofollow"><code>sqlite3</code> driver</a></p>
<p>SQLite integrates seamlessly and is robust, offering a reliable database
schema and validation. Despite utilizing CGO, its seamless operation and
built-in full text search make it indispensable.</p>
<pre><code><span><span><span>package</span> <span>main</span>
</span></span><span><span>
</span></span><span><span><span>import</span> <span>(</span>
</span></span><span><span>    <span>&#34;database/sql&#34;</span>
</span></span><span><span>    <span>&#34;fmt&#34;</span>
</span></span><span><span>    <span>&#34;log&#34;</span>
</span></span><span><span>    <span>_</span> <span>&#34;github.com/mattn/go-sqlite3&#34;</span>
</span></span><span><span><span>)</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>main</span><span>()</span> <span>{</span>
</span></span><span><span>    <span>db</span><span>,</span> <span>err</span> <span>:=</span> <span>sql</span><span>.</span><span>Open</span><span>(</span><span>&#34;sqlite3&#34;</span><span>,</span> <span>&#34;./test.db&#34;</span><span>)</span>
</span></span><span><span>    <span>if</span> <span>err</span> <span>!=</span> <span>nil</span> <span>{</span>
</span></span><span><span>        <span>log</span><span>.</span><span>Fatal</span><span>(</span><span>err</span><span>)</span>
</span></span><span><span>    <span>}</span>
</span></span><span><span>    <span>defer</span> <span>db</span><span>.</span><span>Close</span><span>()</span>
</span></span><span><span>
</span></span><span><span>    <span>statement</span><span>,</span> <span>_</span> <span>:=</span> <span>db</span><span>.</span><span>Prepare</span><span>(</span><span>&#34;CREATE TABLE IF NOT EXISTS people (id INTEGER PRIMARY KEY, firstname TEXT, lastname TEXT)&#34;</span><span>)</span>
</span></span><span><span>    <span>statement</span><span>.</span><span>Exec</span><span>()</span>
</span></span><span><span>
</span></span><span><span>    <span>res</span><span>,</span> <span>_</span> <span>:=</span> <span>db</span><span>.</span><span>Exec</span><span>(</span><span>&#34;INSERT INTO people (firstname, lastname) VALUES (?, ?)&#34;</span><span>,</span> <span>&#34;John&#34;</span><span>,</span> <span>&#34;Doe&#34;</span><span>)</span>
</span></span><span><span>    <span>id</span><span>,</span> <span>_</span> <span>:=</span> <span>res</span><span>.</span><span>LastInsertId</span><span>()</span>
</span></span><span><span>    <span>fmt</span><span>.</span><span>Println</span><span>(</span><span>&#34;Inserted Row ID:&#34;</span><span>,</span> <span>id</span><span>)</span>
</span></span><span><span><span>}</span>
</span></span></code></pre></li>
<li>
<p><a href="https://github.com/onsi/ginkgo" rel="nofollow"><code>ginkgo</code></a></p>
<p>Ginkgo offers a structured, BDD-style TDD framework. It’s readable and,
combined with <code>gomega</code>, provides powerful assertion capabilities.</p>
<pre><code><span><span><span>package</span> <span>main_test</span>
</span></span><span><span>
</span></span><span><span><span>import</span> <span>(</span>
</span></span><span><span>    <span>.</span> <span>&#34;github.com/onsi/ginkgo&#34;</span>
</span></span><span><span>    <span>.</span> <span>&#34;github.com/onsi/gomega&#34;</span>
</span></span><span><span><span>)</span>
</span></span><span><span>
</span></span><span><span><span>var</span> <span>_</span> <span>=</span> <span>Describe</span><span>(</span><span>&#34;Calculator&#34;</span><span>,</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>    <span>Context</span><span>(</span><span>&#34;Addition&#34;</span><span>,</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>        <span>It</span><span>(</span><span>&#34;should add two numbers&#34;</span><span>,</span> <span>func</span><span>()</span> <span>{</span>
</span></span><span><span>            <span>Expect</span><span>(</span><span>1</span> <span>+</span> <span>1</span><span>).</span><span>To</span><span>(</span><span>Equal</span><span>(</span><span>2</span><span>))</span>
</span></span><span><span>        <span>})</span>
</span></span><span><span>    <span>})</span>
</span></span><span><span><span>})</span>
</span></span></code></pre></li>
<li>
<p><a href="https://github.com/golangci/golangci-lint" rel="nofollow"><code>golangci-lint</code></a></p>
<p>This tool has significantly streamlined my coding process, GolangCI-Lint. My
go-to tool for all things linting. GolangCI-Lint doesn’t just stop at
identifying issues; it extends its utility by offering auto-fixes for numerous
detected problems, a feature that has proven invaluable time and again.</p>
<p>Here’s a peek at my default configuration:</p>
<pre><code><span><span><span>run</span><span>:</span><span>
</span></span></span><span><span><span>  </span><span>skip-files</span><span>:</span><span> </span><span>[]</span><span>
</span></span></span><span><span><span></span><span>linters</span><span>:</span><span>
</span></span></span><span><span><span>  </span><span>enable-all</span><span>:</span><span> </span><span>true</span><span>
</span></span></span><span><span><span>  </span><span>disable</span><span>:</span><span>
</span></span></span><span><span><span>    </span>- <span>depguard</span><span>
</span></span></span><span><span><span>    </span>- <span>exhaustruct</span><span>
</span></span></span><span><span><span>    </span>- <span>paralleltest</span><span>
</span></span></span><span><span><span>    </span>- <span>gomoddirectives</span><span>
</span></span></span><span><span><span>    </span>- <span>lll</span><span>
</span></span></span><span><span><span>    </span><span># linters deprecated</span><span>
</span></span></span><span><span><span>    </span>- <span>deadcode</span><span>
</span></span></span><span><span><span>    </span>- <span>exhaustivestruct</span><span>
</span></span></span><span><span><span>    </span>- <span>golint</span><span>
</span></span></span><span><span><span>    </span>- <span>ifshort</span><span>
</span></span></span><span><span><span>    </span>- <span>interfacer</span><span>
</span></span></span><span><span><span>    </span>- <span>nosnakecase</span><span>
</span></span></span><span><span><span>    </span>- <span>maligned</span><span>
</span></span></span><span><span><span>    </span>- <span>scopelint</span><span>
</span></span></span><span><span><span>    </span>- <span>structcheck</span><span>
</span></span></span><span><span><span>    </span>- <span>varcheck</span><span>
</span></span></span></code></pre><p>In this configuration, I’ve activated most linters while deactivating a few
that didn’t align well with my preferences. One particular linter, <code>errcheck</code>,
stands out for its utility. It ensures that all returned errors are aptly
checked with <code>if err != nil {}</code>, helping maintain the codebase’s robustness.</p>
</li>
<li>
<p><a href="https://github.com/samber/lo" rel="nofollow"><code>lo</code></a></p>
<p>This is a utility library for managing different data types in Go. It uses
generics, providing functions similar to those in Ruby’s enumerable library.</p>
<pre><code><span><span><span>package</span> <span>main</span>
</span></span><span><span>
</span></span><span><span><span>import</span> <span>(</span>
</span></span><span><span>  <span>&#34;fmt&#34;</span>
</span></span><span><span>  <span>&#34;github.com/samber/lo&#34;</span>
</span></span><span><span><span>)</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>main</span><span>()</span> <span>{</span>
</span></span><span><span>  <span>slice</span> <span>:=</span> <span>[]</span><span>int</span><span>{</span><span>1</span><span>,</span> <span>2</span><span>,</span> <span>3</span><span>,</span> <span>4</span><span>}</span>
</span></span><span><span>  <span>mappedSlice</span> <span>:=</span> <span>lo</span><span>.</span><span>Map</span><span>(</span><span>slice</span><span>,</span> <span>func</span><span>(</span><span>i</span> <span>int</span><span>)</span> <span>int</span> <span>{</span>
</span></span><span><span>    <span>return</span> <span>i</span> <span>*</span> <span>10</span>
</span></span><span><span>  <span>})</span>
</span></span><span><span>
</span></span><span><span>  <span>fmt</span><span>.</span><span>Println</span><span>(</span><span>mappedSlice</span><span>)</span> <span>// Prints: [10 20 30 40]
</span></span></span><span><span><span></span><span>}</span>
</span></span></code></pre></li>
<li>
<p><a href="https://go.dev/blog/slog" rel="nofollow"><code>log/slog</code></a></p>
<p>This structured logger is a recent addition to the Go standard library,
offering message and parameter logging with exceptional JSON output, enhancing
the coding experience by providing structured, understandable logs.</p>
</li>
</ul>
<p>I am grateful to the developers behind these incredible libraries for their
contribution to the Go community, making the developer experience more pleasant
and efficient.</p>
<p>Having a preferred set of libraries that align with your coding style can
significantly enhance your coding experience, streamline your workflow, and
increase productivity. Each library mentioned has its unique offerings and aids
in creating efficient and clean code. Whether you are a seasoned developer or a
beginner exploring the Go programming landscape, I hope you find these libraries
as helpful as I have.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-09-30.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Seeking Optimal SQL Patterns with Go</title>
      <link>https://jtarchie.com/posts/2023-09-21.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="seeking-optimal-sql-patterns-with-go"><a href="#seeking-optimal-sql-patterns-with-go" rel="nofollow">#</a> Seeking Optimal SQL Patterns with Go</h1>
<p>One of the challenges software engineers face when working with SQL in Go is
managing boilerplate. It can be cumbersome and repetitive, detracting from the
application’s core logic. Recently, I’ve embarked on a journey exploring <code>sqlc</code>,
a tool designed to generate type-safe code directly from SQL, seemingly
promising an efficient way to handle SQL in Go.</p>
<p>The <code>sqlc</code> tool is available on <a href="https://github.com/sqlc-dev/sqlc" rel="nofollow">GitHub</a>, and
my initial impression was positive. It offered an appealing solution for many of
the standard SQL boilerplate in Go.</p>
<p>You can provide <code>*.sql</code> files to <code>sqlc</code> to generate the code. This is great
because copying and pasting SQL queries to and from a database UI is pretty
simple. <code>sqlc</code> requires meta information in the form of comments before each
query.</p>
<p>For example, this query would get value from a table:</p>
<pre><code><span><span><span>-- name: Get :one
</span></span></span><span><span><span></span><span>SELECT</span><span> </span><span>value</span><span>
</span></span></span><span><span><span></span><span>FROM</span><span> </span><span>keys</span><span>
</span></span></span><span><span><span></span><span>WHERE</span><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>@</span><span>name</span><span>;</span><span>
</span></span></span></code></pre><p>The <code>name</code> specifies how to name the function that <code>sqlc</code> will generate. The
<code>:one</code> directive says how many rows are returned from the query. You can see
more in the
<a href="https://play.sqlc.dev/p/be794afff24aaa250f2ab773b80bf4625def2b18d4cd564adf6e4a25ff8079b5" rel="nofollow"><code>sqlc</code> playground</a>
of this directive affect the code generation.</p>
<p>However, as with any tool, the more I delved into its capabilities, the more
nuances I discovered. It became evident that while <code>sqlc</code> is optimized for
PostgreSQL, it supports MySQL and SQLite, but not some advanced features (like
JSON).</p>
<p>I began to face challenges with <code>sqlc</code> when building queries that leveraged
sophisticated database features, particularly those related to JSON
manipulation. In these scenarios, <code>sqlc</code> couldn’t always maintain its contract
of generating the corresponding Go boilerplate. This limitation meant reverting
to manual code-writing for specific functions.</p>
<p>For example, the query that required custom generation:</p>
<pre><code><span><span><span>-- name: ListRange :many
</span></span></span><span><span><span></span><span>	</span><span>SELECT</span><span> </span><span>json_each</span><span>.</span><span>value</span><span>
</span></span></span><span><span><span>	</span><span>FROM</span><span> </span><span>keys</span><span>,</span><span>
</span></span></span><span><span><span>		</span><span>json_each</span><span>(</span><span>keys</span><span>.</span><span>value</span><span>)</span><span>
</span></span></span><span><span><span>	</span><span>WHERE</span><span> </span><span>keys</span><span>.</span><span>name</span><span> </span><span>=</span><span> </span><span>?</span><span>1</span><span>
</span></span></span><span><span><span>	</span><span>AND</span><span> </span><span>json_each</span><span>.</span><span>key</span><span> </span><span>&gt;=</span><span> </span><span>IIF</span><span>(</span><span>?</span><span>2</span><span> </span><span>&gt;=</span><span>0</span><span>,</span><span> </span><span>?</span><span>2</span><span>,</span><span> </span><span>json_array_length</span><span>(</span><span>keys</span><span>.</span><span>value</span><span>)</span><span> </span><span>+</span><span> </span><span>?</span><span>2</span><span>)</span><span>
</span></span></span><span><span><span>	</span><span>AND</span><span> </span><span>json_each</span><span>.</span><span>key</span><span> </span><span>&lt;=</span><span> </span><span>IIF</span><span>(</span><span>?</span><span>3</span><span> </span><span>&gt;=</span><span>0</span><span>,</span><span> </span><span>?</span><span>3</span><span>,</span><span> </span><span>json_array_length</span><span>(</span><span>keys</span><span>.</span><span>value</span><span>)</span><span> </span><span>+</span><span> </span><span>?</span><span>3</span><span>);</span><span>
</span></span></span></code></pre><p>Inspired by <code>sqlc</code>’s premise, I considered creating my code generation. The goal
was to automate the process as much as possible, allowing me to focus on the SQL
logic. One of the valuable features <code>sqlc</code> offered was its ability to infer
return types and handle type casting automatically for specific queries. In my
code generation attempts, I wanted to integrate this feature, and thus, I
explored leveraging ChatGPT for code generation, specifically to infer return
types.</p>
<p>For example, give the following SQL query:</p>
<pre><code><span><span><span>-- name: ListLength :one
</span></span></span><span><span><span></span><span>SELECT</span><span> </span><span>json_array_length</span><span>(</span><span>value</span><span>)</span><span>
</span></span></span><span><span><span></span><span>FROM</span><span> </span><span>keys</span><span>
</span></span></span><span><span><span></span><span>WHERE</span><span> </span><span>name</span><span> </span><span>=</span><span> </span><span>?</span><span>1</span><span>
</span></span></span></code></pre><p><code>sqlc</code> could not infer the data type given from <code>json_array_length</code> function.
(Related to the before-mentioned missing JSON integration.) With ChatGPT, I
prompted,
<code>Please fix the return type of this Go function that is an SQL query wrapper so that it returns the correct type.</code></p>
<p>So this:</p>
<pre><code><span><span><span>func</span> <span>(</span><span>q</span> <span>*</span><span>Queries</span><span>)</span> <span>ListLength</span><span>(</span><span>ctx</span> <span>context</span><span>.</span><span>Context</span><span>,</span> <span>name</span> <span>string</span><span>)</span> <span>(</span><span>interface</span><span>{},</span> <span>error</span><span>)</span> <span>{</span>
</span></span><span><span>	<span>row</span> <span>:=</span> <span>q</span><span>.</span><span>queryRow</span><span>(</span><span>ctx</span><span>,</span> <span>q</span><span>.</span><span>listLengthStmt</span><span>,</span> <span>listLength</span><span>,</span> <span>name</span><span>)</span>
</span></span><span><span>	<span>var</span> <span>json_array_length</span> <span>interface</span><span>{}</span>
</span></span><span><span>	<span>err</span> <span>:=</span> <span>row</span><span>.</span><span>Scan</span><span>(</span><span>&amp;</span><span>json_array_length</span><span>)</span>
</span></span><span><span>	<span>return</span> <span>json_array_length</span><span>,</span> <span>err</span>
</span></span><span><span><span>}</span>
</span></span></code></pre><p>Became:</p>
<pre><code><span><span><span>func</span> <span>(</span><span>q</span> <span>*</span><span>Queries</span><span>)</span> <span>ListLength</span><span>(</span><span>ctx</span> <span>context</span><span>.</span><span>Context</span><span>,</span> <span>name</span> <span>string</span><span>)</span> <span>(</span><span>int64</span><span>,</span> <span>error</span><span>)</span> <span>{</span>
</span></span><span><span>	<span>row</span> <span>:=</span> <span>q</span><span>.</span><span>queryRow</span><span>(</span><span>ctx</span><span>,</span> <span>q</span><span>.</span><span>listLengthStmt</span><span>,</span> <span>listLength</span><span>,</span> <span>name</span><span>)</span>
</span></span><span><span>	<span>var</span> <span>json_array_length</span> <span>int64</span>
</span></span><span><span>	<span>err</span> <span>:=</span> <span>row</span><span>.</span><span>Scan</span><span>(</span><span>&amp;</span><span>json_array_length</span><span>)</span>
</span></span><span><span>	<span>return</span> <span>json_array_length</span><span>,</span> <span>err</span>
</span></span><span><span><span>}</span>
</span></span></code></pre><p>The results were interesting. ChatGPT was indeed able to modify the generated
code to determine return types. However, the speed left much to be desired.
Using ChatGPT for code generation in real-time proved too slow for my needs,
elongating the feedback loop significantly.</p>
<p>Despite this setback, my exploration didn’t end there. I further expanded my
investigations into integrating ChatGPT with SQL queries to Go boilerplate
conversion, culminating in a tool found
<a href="https://github.com/jtarchie/sqlc-gpt" rel="nofollow">here</a>. This is going to require its own
post because of the depth of the prompt. It works, but again, it needs to be
faster to be a viable alternative for a fast feedback loop.</p>
<p>While my exploration with <code>sqlc</code> and Go was enlightening, my interest in
efficient SQL handling naturally led me to consider other languages and tools.
Rust’s <code>sqlx</code> caught my attention. This Rust SQL toolkit offers compile-time
checked queries without a DSL, supporting a range of databases, which can be
explored <a href="https://github.com/launchbadge/sqlx#sqlx-is-not-an-orm" rel="nofollow">here</a>. I’ll be
working with this tool to learn Rust, too.</p>
<p>Tools like <code>sqlc</code> hold great promise, but they also underscore the complexities
of integrating SQL and programming languages. As the landscape of software
development tools evolves, I remain excited and optimistic about the future
possibilities for efficient database management and integration.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-09-21.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Exploring Go with CGO Overhead and `sqlc`</title>
      <link>https://jtarchie.com/posts/2023-09-20.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="exploring-go-with-cgo-overhead-and-sqlc"><a href="#exploring-go-with-cgo-overhead-and-sqlc" rel="nofollow">#</a> Exploring Go with CGO Overhead and <code>sqlc</code></h1>
<p>In a recent endeavor, I’ve been tinkering with building a server in Golang. It’s
built around a low-level TCP protocol, each connection sparking its goroutine.
What makes this interesting is using cgo to interface with a library. As many
Golang fans might know, each cgo call incurs a time penalty due to the
transition from C to Go. This bridge had me curious - has this ever been a
performance bottleneck for others?</p>
<p>To paint a clearer picture, I stumbled upon
<a href="https://shane.ai/posts/cgo-performance-in-go1.21/" rel="nofollow">Shane’s article</a> discussing
the performance of CGO in Go 1.21. A quick synopsis: cgo calls roughly take
about 40ns – on par with the time <code>encoding/json</code> spends parsing a single-digit
integer. This performance scales with a core count of about 16 cores, after
which contention issues slow down the process. It’s worth noting that while
Shane suggests cgo performance might be satisfactory, it’s not a universal
endorsement of using cgo. A 20-core machine was his testing playground, after
all.</p>
<p>Jokes about nanosecond optimizations aside, I considered moving my inner loop to
Rust or C. If the cgo boundary is crossed frequently, it’s worth contemplating
the potential performance gains. Plus, learning Rust has been on my bucket list.</p>
<p>Another intriguing aspect of my journey was discovering that the library I’m
interfacing with might involve more than just one cgo call. Dive deeper? Well,
that’s on my to-do list.</p>
<p>I also dabbled with the Go runtime settings in the name of optimization. I
naively considered setting <code>GOMAXPROCS=1</code> after removing a mutex, expecting Go’s
runtime to ensure sequential operation. To my surprise, putting it to
<code>GOMAXPROCS=2</code> performed better. But, a word of caution: trusting the runtime as
its mutex beyond this point led to a drop in performance.</p>
<p>Integrating SQLite through the
<a href="https://github.com/mattn/go-sqlite3" rel="nofollow">CGO library</a> added another layer of depth
to my experiment. Here’s the gist of my learnings:</p>
<ul>
<li>Using <code>db/sql</code>, you can open multiple database connections using a mutex.</li>
<li>SQLite has distinct methods to lock files, transactions, and operations (using
mutex).</li>
<li>Setting the max open connections to 1 and modifying SQLite settings can boost
performance by 2-3 times. However, you can’t simultaneously access the
database file with CLI SQLite3.</li>
</ul>
<p>And for those curious about the code, feel free to dive into
<a href="https://github.com/jtarchie/sqlettus" rel="nofollow">sqlettus on GitHub</a>.</p>
<p>As the project evolved, my vision crystallized. The aim? Building something
compatible with the Redis protocol but backed by databases. Why? With its
transaction capabilities and established protocols, SQL is the perfect match.
Besides, being able to inspect your database through both SQL and the Redis
protocol has its perks.</p>
<p>Using <code>sqlc</code> in the Golang project was beneficial. Its purpose is to remove the
need for boilerplate code when writing, executing, and returning the values of
SQL queries. It was great until it wasn’t.</p>
<p>It supports Postgres, MySQL, and SQLite databases. It seems, however, to support
Postgres features. I’ve been using SQLite for the low barrier to entry, and it
supports advanced JSON features. <code>sqlc</code> did not like the usage of <code>json_each</code>,
<code>json_value</code>, etc. in some places, however.</p>
<p>My only workaround for more complex queries was to write the boilerplate code I
was trying to avoid.</p>
<p>Here’s a sneak peek into my progress:</p>
<ul>
<li>Built a router to handle TCP directives.</li>
<li>Emphasized handling tasks at the SQL level, harnessing <code>sqlc</code> to cut down on
boilerplate (unearthed a few bugs in the process!).</li>
<li>Continual refactoring to make adding directives a breeze.</li>
<li>Hunting for a consistent pattern for various database backends.</li>
</ul>
<p>As a bonus, I’ve received some great suggestions, like renaming it to
<code>sqlettuce.</code> And yes, spelling can be a challenging game!</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-09-20.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Using Go Generics for Function Caching</title>
      <link>https://jtarchie.com/posts/2023-06-14.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="using-go-generics-for-function-caching"><a href="#using-go-generics-for-function-caching" rel="nofollow">#</a> Using Go Generics for Function Caching</h1>
<p>I’m currently engaged in a side project where I’m exploring web application
development using Golang. As part of this endeavor, I’m incorporating elements
from other web frameworks like Ruby on Rails to handle server-side HTML
rendering, database connection management, form handling, and more.</p>
<p>During the course of my work, I decided to dive into caching and wanted to
experiment with different strategies. One of the advantages of Golang over other
languages is its compilation and support for multi-threading through go
routines. I wanted to implement executable caching without relying on external
tools like Redis or Memcached.</p>
<p>There are various patterns for caching, often revolving around the concept of a
key-value store. To summarize it in haiku form:</p>
<blockquote>
<p>Here is my data, It is named this way, give it back, I don’t care how.</p>
</blockquote>
<p>In Golang, there are several in-process key-value stores available. One popular
choice is <a href="https://github.com/boltdb/bolt" rel="nofollow">BoltDB</a> due to its reliability and
battle-tested nature. However, upon careful consideration, it still felt too
similar to Redis. I was inclined to approach this challenge more
programmatically, perhaps even writing the caching mechanism myself.</p>
<p>Caching the return value of a function call requires knowledge of the types in
advance, so that the value can be appropriately cached.</p>
<p>This is a short example of the ergonomics I was aiming for.</p>
<pre><code><span><span><span>package</span> <span>main</span>
</span></span><span><span>
</span></span><span><span><span>func</span>  <span>SomeFunc</span><span>()</span> <span>int</span> <span>{</span>
</span></span><span><span>  <span>// complex operation
</span></span></span><span><span><span></span>  <span>return</span>  <span>1</span>
</span></span><span><span><span>}</span>
</span></span><span><span>
</span></span><span><span><span>func</span>  <span>CacheWrapper</span><span>(</span><span>fun</span> <span>func</span><span>()</span> <span>int</span><span>)</span> <span>func</span><span>()</span> <span>int</span> <span>{</span>
</span></span><span><span>  <span>var</span>  <span>cached</span> <span>*</span><span>int</span>
</span></span><span><span>  <span>return</span>  <span>func</span><span>()</span> <span>int</span> <span>{</span>
</span></span><span><span>    <span>if</span> <span>cached</span> <span>==</span> <span>nil</span> <span>{</span>
</span></span><span><span>      <span>value</span> <span>:=</span> <span>fun</span><span>()</span>
</span></span><span><span>      <span>cached</span> <span>=</span> <span>&amp;</span><span>value</span>
</span></span><span><span>    <span>}</span>
</span></span><span><span>    <span>return</span> <span>*</span><span>cached</span>
</span></span><span><span>  <span>}</span>
</span></span><span><span><span>}</span>
</span></span><span><span>
</span></span><span><span><span>func</span>  <span>main</span><span>()</span> <span>{</span>
</span></span><span><span>  <span>invoke</span> <span>:=</span> <span>CacheWrapper</span><span>(</span><span>SomeFunc</span><span>)</span>
</span></span><span><span>  <span>invoke</span><span>()</span> <span>// initial call
</span></span></span><span><span><span></span>  <span>invoke</span><span>()</span> <span>// using cached value
</span></span></span><span><span><span></span><span>}</span>
</span></span></code></pre><p>As you can see <code>CachedWrapper</code> is ensuring that the expensive function is only
called once. This Just Works™ with <code>int</code> return types, however. With early
Golang libraries, I’d have to create a <code>CachedWrapper</code> for each return type I’d
ever want to use.</p>
<p>With more recent versions of Golang, however, generics were introduces to allow
types to be abstracted away. The type could be inferred at compiled time based
on the return type.</p>
<p>his is short example of the ergonomics I was aiming for. You’ll see the return
type referenced as <code>R</code> in <code>CacheWrapper</code> the function declaration.</p>
<pre><code><span><span><span>package</span> <span>main</span>
</span></span><span><span>
</span></span><span><span><span>func</span>  <span>SomeFunc</span><span>()</span> <span>int</span> <span>{</span>
</span></span><span><span>  <span>// complex operation
</span></span></span><span><span><span></span>  <span>return</span>  <span>1</span>
</span></span><span><span><span>}</span>
</span></span><span><span>
</span></span><span><span><span>func</span> <span>CacheWrapper</span><span>[</span><span>R</span> <span>any</span><span>](</span><span>fun</span> <span>func</span><span>()</span> <span>R</span><span>)</span> <span>func</span><span>()</span> <span>R</span> <span>{</span>
</span></span><span><span>  <span>var</span>  <span>cached</span> <span>*</span><span>R</span> <span>// R represent the return function value
</span></span></span><span><span><span></span>  <span>return</span>  <span>func</span><span>()</span> <span>R</span> <span>{</span>
</span></span><span><span>    <span>if</span> <span>cached</span> <span>==</span> <span>nil</span> <span>{</span>
</span></span><span><span>      <span>value</span> <span>:=</span> <span>fun</span><span>()</span>
</span></span><span><span>      <span>cached</span> <span>=</span> <span>&amp;</span><span>value</span>
</span></span><span><span>    <span>}</span>
</span></span><span><span>    <span>return</span> <span>*</span><span>cached</span>
</span></span><span><span>  <span>}</span>
</span></span><span><span><span>}</span>
</span></span><span><span>
</span></span><span><span><span>func</span>  <span>main</span><span>()</span> <span>{</span>
</span></span><span><span>  <span>invoke</span> <span>:=</span> <span>CacheWrapper</span><span>(</span><span>SomeFunc</span><span>)</span>
</span></span><span><span>  <span>invoke</span><span>()</span> <span>// initial call
</span></span></span><span><span><span></span>  <span>invoke</span><span>()</span> <span>// using cached value
</span></span></span><span><span><span></span><span>}</span>
</span></span></code></pre><p><em>NOTE</em>: This is caching of a function call, not memoization. If we were
memoizing, we’d be taking arguments into account. The caching would be more
complex. Here we are just considering a function is called, and the returned
value is essential.</p>
<p>With <code>R</code> as a placeholder, we can use <code>any</code> type as the function’s return value.
It is inferred from the <code>fun</code> return type, which allows us never to have to know
about it ahead of time explicitly. The compiler does the work for us.</p>
<p>I can use this in many places with minimal intrusion into my current codebase. I
have to import the <code>CacheWrapper</code> function and use it. Generics in Go are
fantastic and save me so much code!</p>
<p>Some next steps for ensuring caching functionality would be expiration and
thread safeness. I’ll follow up with a continuation or a library.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-06-14.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Balancing Website with Limited Access Hours</title>
      <link>https://jtarchie.com/posts/2023-05-12.html</link>
      <description>This examines the idea of limited website access hours, questioning the necessity of 24/7 availability. It highlights potential benefits such as resource optimization, enhanced user experiences, mindful internet use, improved security, and reduced burnout among web professionals, encouraging a reevaluation of website accessibility for a balanced digital landscape.</description>
      <content:encoded><![CDATA[<h1 id="balancing-website-with-limited-access-hours"><a href="#balancing-website-with-limited-access-hours" rel="nofollow">#</a> Balancing Website with Limited Access Hours</h1>
<p>The digital age has transformed how we access information, services, and
products, with websites becoming an essential part of our daily lives.
Round-the-clock availability has been a defining characteristic of the online
landscape, but is 24/7 access necessary for all websites? In this blog post,
we’ll explore the arguments for limited access hours for websites akin to the
operating hours of traditional brick-and-mortar businesses and consider the
potential benefits of such a model.</p>
<p>Redefining the Online Experience</p>
<ol>
<li>Resource Management and Cost Efficiency</li>
</ol>
<p>By adopting traditional operating hours, websites can allocate resources more
effectively, concentrating efforts during peak periods and potentially reducing
costs associated with 24/7 operations. Limited access hours could lead to
decreased server demands and energy consumption, resulting in cost savings and
contributing to a more sustainable digital environment.</p>
<ol>
<li>Enhancing User Experience and Fostering Exclusivity</li>
</ol>
<p>Limited website access could create a more focused and immersive experience for
users, offering a refreshing change from the constant barrage of online
information. Controlled access also fosters a sense of exclusivity and
anticipation, encouraging users to engage more intentionally with websites
during designated periods.</p>
<ol>
<li>Encouraging Mindful Internet Use and Balancing Online-Offline Lives</li>
</ol>
<p>With the ubiquity of the internet, society has become increasingly dependent on
digital services. Limited website access serves as a reminder to strike a
balance between our online and offline lives, prompting individuals to manage
their time effectively, ensuring that they are not overly reliant on constant
connectivity, and allowing opportunities for offline pursuits and face-to-face
interactions.</p>
<ol>
<li>Addressing Security, Maintenance, and Content Quality Concerns</li>
</ol>
<p>Limited access hours offer a window for critical maintenance tasks, security
checks, and system upgrades without causing significant disruptions to users.
This allows businesses to ensure their websites are robust, secure, and
up-to-date while minimizing the impact on the overall user experience.
Additionally, website owners can prioritize quality over quantity by shifting
focus from constant updates to creating and curating meaningful content.</p>
<ol>
<li>Supporting Web Professionals and Reducing Burnout</li>
</ol>
<p>Operating a website 24/7 requires a team of dedicated professionals to monitor,
update, and troubleshoot potential issues constantly. Limiting website access
hours could alleviate pressure on these individuals, allowing for a better
work-life balance and potentially reducing burnout.</p>
<p>As the digital realm continues to evolve, it is crucial to reevaluate
traditional assumptions about website accessibility. By embracing limited
website access hours, we can balance constant availability and intentional,
focused access, offering potential benefits such as resource optimization,
enhanced user experiences, and a healthier relationship with technology. By
rethinking website accessibility, we can pave the way for a more balanced and
sustainable digital landscape that caters to the diverse needs of businesses and
users alike.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-05-12.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Communication &gt;&gt; Process &gt;&gt; Tools</title>
      <link>https://jtarchie.com/posts/2023-04-13.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="communication--process--tools"><a href="#communication--process--tools" rel="nofollow">#</a> Communication » Process » Tools</h1>
<p>When working in a team or an organization with multiple teams, finding effective
ways to work with each other is always a high priority. We want to be effective
and accomplish something, which is what modern skill-based work is all about. In
my experience as a software engineer, I’ve learned that communication is vital.
This isn’t a lost art or a revolutionary idea but rather a foundational concept
for my discussion.</p>
<p>Communication can be one-on-one, between teams, or even within a team. There are
many methods of communication, such as video calls, meetings, sharing documents,
or screencasts. However, with so many communication methods, it can sometimes
become overwhelming. Overcommunication is only occasionally effective
communication. This is where the importance of having a process comes into play.</p>
<p>Process for communication can streamline interactions, such as setting up
meetings or establishing protocols for discussing priorities. Communication
becomes more efficient when all team members are on the same page. However, at
any scale, processes can also reach a point where they become ineffective. Tools
such as story trackers, Gantt charts, and other ways to view progress can be
introduced to complement the process.</p>
<p>For example, a director of engineering may want to see the progress of the ten
teams under them to determine if they will meet quarterly goals. Each team has a
view to see where they’ve progressed along the quarterly goals, but the director
may not need to see the granular breakdown. Tools can help communicate through a
process, streamlining and providing both the eagle eye and the more broken-down
view.</p>
<p>Throughout my career, I have worked in various environments, from startups to
enterprise companies. I have observed that when communication and process are
not in place, tools are often seen as a solution. For instance, I once worked at
a startup struggling to find a project management tool. They had tried five
different tools, none of which fit their needs. The issue was that they needed a
process for the tools to complement, which led to a constant search for the
perfect tool instead of addressing the underlying problems. This experience made
me realize the importance of having a solid communication foundation and
well-defined processes before introducing tools.</p>
<p>In another example, a larger company used Jira, a popular project management
tool, but needed an established process for tracking progress. They tried to use
Jira’s advanced features to implement a process, but they were still struggling
a year and a half later. Clearly, the need for a tool had not arisen from an
existing process or communication structure, which led to difficulties in making
the tool work effectively.</p>
<p>To further emphasize the importance of effective communication, consider the
following benefits:</p>
<ul>
<li>Improved collaboration: Strong communication within a team or organization
leads to better collaboration among team members. This, in turn, can result in
more innovative solutions and higher-quality output.</li>
<li>Enhanced decision-making: When information is shared openly and effectively,
team members can make more informed decisions that align with the
organization’s objectives.</li>
<li>Increased trust and morale: Open and transparent communication fosters trust
and confidence among team members, resulting in a more positive and productive
work environment.</li>
</ul>
<p>To improve communication and processes within your team or organization,
consider implementing the following best practices:</p>
<ul>
<li>Establish clear communication channels: Ensure everyone knows the appropriate
channels for sharing information and raising concerns. This can include
regular team meetings, designated communication platforms, or established
protocols for escalating issues.</li>
<li>Set expectations: Clearly define roles, responsibilities, and expectations for
team members to ensure everyone understands their part in the larger
organizational goals.</li>
<li>Encourage open dialogue: Foster an environment where team members feel
comfortable sharing ideas, concerns, and feedback. This can lead to more
innovative solutions and more robust team dynamics.</li>
<li>Regularly review and adjust processes: Periodically evaluate the effectiveness
of your team’s communication processes and make adjustments as necessary. This
can help keep communication channels clear and ensure everyone works
efficiently.</li>
<li>Provide training and support: Offer training and support to team members on
effective communication techniques and any tools or processes being used
within the organization. This can help ensure everyone is on the same page and
using the available resources effectively.</li>
<li>Recognize and celebrate success: Acknowledge and celebrate the accomplishments
of your team members. This can help build a sense of camaraderie and encourage
continued effective communication and collaboration.</li>
<li>Lead by example: As a leader, set the tone for open and effective
communication by being transparent, honest, and supportive in your
interactions with your team members. This can create a positive work
environment that fosters strong communication and collaboration.</li>
</ul>
<p>In conclusion, communication is a critical foundation for effective teamwork and
organizational success. It is essential to prioritize communication before
process and process before tools. Establishing clear communication channels,
setting expectations, and providing support can create an environment where
teams can work together more efficiently and effectively. Remember that solid
communication leads to better collaboration, enhanced decision-making, and
increased trust and morale within your team or organization.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-04-13.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Programming Journey</title>
      <link>https://jtarchie.com/posts/2023-04-06.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="programming-journey"><a href="#programming-journey" rel="nofollow">#</a> Programming Journey</h1>
<p>I want to share the story of how I got into software engineering. I aim to
present an autobiographical account while acknowledging that we all come from
different backgrounds. One of the things I’ve always respected about working
with people is the diversity of experiences we bring to the table. Sometimes,
you need to know where you started to understand where you are.</p>
<p>My first encounter with a computer was an Apple IIe – we’re talking old-school
here. It used a giant floppy disk you had to insert for anything to load. It
didn’t have anything on it. As a kid, I didn’t know how to use it. This machine
was more about pressing buttons and moving the mouse – although, come to think
of it, I don’t believe it had a mouse.</p>
<p>My first real exposure to a usable computer was a 486SX. It had Windows 3.1
installed and provided access to the DOS command line prompt. Growing up in Hong
Kong, we had access to more programs than most people in the United States would
have. I won’t go into detail about that, but with that access came exposure to
the various capabilities of the system. From games to drawing programs, I was
fascinated by the interactivity of this machine.</p>
<p>I was interested in how it all started through discovery rather than mere
exposure. I knew from using the DOS command line that you could run programs
with .exe extensions, and the ones that didn’t run but could be read were text
files. You could load these files into the editing program. It was just plain
text, similar to how we can pack things in Notepad, WordPad, or our favorite
text editor. When I loaded one of the executables into the editor, something
strange happened – it would be filled with all these characters I didn’t
recognize. We now know them as extended ASCII characters, but they were a
mystery to me then. What was so intriguing was that, since my only comparison
was a text file, I wondered how people could sit there and edit executables
using these extended ASCII characters. How did it work? How did people decipher
what was happening?</p>
<p>In middle school, I discovered that one of my friend’s dads was a computer
programmer. At that time, I couldn’t tell you exactly what he did; as a kid, I
never probed further to find out. But one day, in the elevator of our apartment
building, when his dad was present, I asked, “Hey, I load these executable files
into the editing program on DOS, and I see how things are written, but I don’t
understand how someone could understand that.” He chuckled and said, “That’s not
how that works,” and then essentially made an offer that changed my life. He
invited me to their apartment, saying, “In an hour, come up, and I’ll show you
how we do it.”</p>
<p>He spent an hour introducing me to QBasic, a programming language that ran on
DOS. He demonstrated a simple program, compiled it, and showed me the
instruction set for setting variables, managing strings, getting input from a
user, displaying something on the screen, and, I believe, even using an if-else
statement. It was one of those moments when someone showed me something, and
everything just clicked. It didn’t mean I fully understood, but the concepts and
discussion were like puzzle pieces that fit together – I just hadn’t completed
the puzzle then. I’m unsure if that’s an apt metaphor, but that’s how it felt.</p>
<p>Over the next few years in middle school, given the limited information and
resources I had access to, I absorbed as much as I could about programming.
Remember, when internet access primarily came through terminals and text-based
browsers like Lynx, I mostly tinkered on my own.</p>
<p>At that time, I was exposed to other things piqued my interest in software.
Bulletin board systems (BBS) that used dial-up modems were prevalent. This
allowed you to connect to someone’s computer, where they might have games to
play, messages you could leave, or even downloads. At some point, there was even
an email-based system called FidoNet that allowed messages to be passed through
multiple bulletin boards. Naturally, I wanted to create my own BBS, but I had no
idea how. Although I wasn’t successful in creating one due to the lack of
accessible knowledge, I did get really good at playing Legends of the Red
Dragon, the best BBS game ever.</p>
<p>Our school introduced a learning tool called HyperStudio Max. It is an
interactive presentation program. It allowed you to create slides or “cards”
with content, and you could have buttons to navigate between slides. It needed
to be more advanced to develop full-fledged games. Still, it offered enough
interactivity for middle schoolers to be productive and learn some basic
programming concepts.</p>
<p>What really helped me in middle school, around eighth grade or the summer after,
was when my parents allowed me to attend a computer summer camp. Besides being
introduced to Monty Python, I learned a new programming language called Pascal.
Like QBasic, it covered the same concepts of logic, inputs, and outputs. Still,
it introduced more recent concepts of code organization, data types, and access
to the display driver. Someone showed me how to draw lines and circles and
animation techniques. I was blown away by the realization that we didn’t have to
write text line by line. I also got introduced to Doom, Quake, and Doom 3 mods.</p>
<p>Being exposed to other people with the same interests opened the door for me,
showing that I could continue to expand upon this. My next level of exposure
came from a book I found at Barnes &amp; Noble in San Francisco. The book, with a
camel on the cover, was about Perl and written by Larry Wall. All the other
programming languages I had learned before were quite simple but didn’t offer
much room for minimizing or optimizing the code – I suppose “verbose” would be
the right word.</p>
<p>The book I picked showed me that a language could be simple yet allow you to
perform complex tasks with minimal code. I know Perl has a bad reputation for
being difficult to understand. Still, this book introduced me to powerful
concepts that I could never return to less versatile languages. Something that
took just one line in Perl would require 30 lines in QBasic. Let’s be honest –
regular expressions (regex) are incredibly powerful.</p>
<p>Perl was also my first introduction to web programming with CGI. Around this
time, Netscape Navigator was a popular browser, and people were writing programs
for this interface. Perl, as a platform, had CPAN for downloading modules, so
you didn’t have to write all the code yourself. The CGI module was just there.</p>
<p>To learn some of these more complex programs for the web, I started by emulating
other people’s programs. Open source was a thing, but it wasn’t as accessible as
now with platforms like GitHub. There were scripts and programs people could use
on their websites, like guestbooks or web forums, which I tried to build myself.
Much of the code I emulated came from a website called Matt’s Script Archive – a
piece of internet history worth checking out.</p>
<p>In my sophomore year of high school, I started taking all the programming
classes I could, including IB Tech and AP Computer Science. I was the only
sophomore to take that class at the time. It focused on C++, and while I
understood the language in terms of syntax, I didn’t grasp how widely used and
versatile it was. I didn’t understand the difference between a compiled language
like C++ and an interpreted language like Perl. I knew I could do the same
things in both, and one allowed me to write code more quickly. For the most
part, the runtime of the programs I built, whether in C++ or Perl, was about the
same, as I needed to do something computationally complex enough to push the
limits of the code.</p>
<p>Since I had Netscape and such, my nerdom and geekdom became cemented as part of
my life. My time in high school was spent on the internet, consuming programming
knowledge and tinkering. I was up until 2 AM every night trying to figure out
how something worked.</p>
<p>As the web became more popular, I started building not necessarily more complex
things but projects that I would use more often on the web, including websites.
Like many high schoolers at the time, I built websites for people trying to get
on the internet to represent a business or have a presence. JavaScript had come
out, showing that you could have minimal interactivity with websites. Still,
even that tiny amount of interactivity was enough to create new things. My
skills expanded when I learned to build my first HTML/JavaScript-based game,
Dynamic HTML. It was a simple two-player game on one computer. Still, it was
much easier than writing console programs, which spoke to the power and
flexibility of web-based languages.</p>
<p>I built a series of themed web pages in my later high school years. One was a
site that a few friends and I created to post weird things, which we thought was
probably the equivalent of internet graffiti. Nothing terribly exciting, just
high schoolers and nerds being weird.</p>
<p><img src="/assets/2023-04-05-thealienz.png" alt="Wayback Machine of thealienz.com"/></p>
<p>Another site I built personally, and I was really into the X-Files then, was a
UFO community website. The original idea was to invite other UFO websites to
join and become their host, creating one central place for people to visit. It
was still a bunch of high schoolers, and no one had any objective evidence.
Everyone was copying and pasting pictures they found from other places on the
internet of aliens and UFOs. Still, it was a good learning experience on how to
host websites, build a community, and share a general interest with people.</p>
<p>The third website I built was a fan website for South Park, which started in
1997, though I didn’t have access until 1998. This website was called
<a href="http://killkenny.com" rel="nofollow">killkenny.com</a>, and we had three sections that made us an
actual webpage. First, we had a hate mail section because we would get hate mail
from people who found South Park disgusting and somehow found our website to
complain about it. Others argued that we should give up the domain for Kilkenny,
Ireland, which is not spelled the same. Another section we had was weekly
trivia. Every time we watched an episode, we noticed that the creators of South
Park had put in little tidbits that would go by quickly, so we would have trivia
questions based on these details. The third section of the website was hosting
the episodes before streaming was a thing. The only real contender in the space
was a program called RealVideo, which had a low bitrate that made the files
small enough for people with only modem connections.</p>
<p><img src="/assets/2023-04-05-killkenny.png" alt="Wayback Machine of killkenny.com"/></p>
<p>I learned a lot from building this website. I was still in Hong Kong then, and
only a few people in Hong Kong could watch South Park or even understand the
cultural references. But I designed, built, and maintained that website for
three years and through a couple of iterations. I never thought much of it since
I was in college and didn’t keep it up as an adult. However, when I discussed
with some people my age about having had a South Park fan website called
<a href="http://killkenny.com" rel="nofollow">killkenny.com</a>, many said they used to download episodes
from there. This showed me the internet’s power, being continents apart and
having people use my site.</p>
<p>So, there you have it - my adventure through programming and how it all began.
From playing with QBasic to building websites and nurturing online communities,
every lesson and experience has shaped who I am as a programmer and person. It
shows how the constantly changing world of computer programming opens up endless
possibilities for those who dare to dive in and create something amazing.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-04-06.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Debugging and Resolving Issues with Go and the `sqlite3` Library</title>
      <link>https://jtarchie.com/posts/2023-03-29.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="debugging-and-resolving-issues-with-go-and-the-sqlite3-library"><a href="#debugging-and-resolving-issues-with-go-and-the-sqlite3-library" rel="nofollow">#</a> Debugging and Resolving Issues with Go and the <code>sqlite3</code> Library</h1>
<h2 id="how-the-problem-started"><a href="#how-the-problem-started" rel="nofollow">#</a> How the problem started</h2>
<p>I recently embarked on a journey to incorporate geographical data into a Golang
program. This was an exciting exploration, as most solutions require PostgreSQL
and PostGIS. However, for the project I’m working on, having a whole database
would add an extra step in the deployment process. I decided that the
application I’m building should either be a static website or have a self-hosted
option for searching for geographical information.</p>
<p>Let’s start by defining the data schema. I have a bunch of latitude and
longitude coordinates, along with a title for each location. For example, store
location 45 with a phone number (123) 555-1234. The objective is to build a
store locator so that when you visit this company’s website, you can input your
location, say, Denver, Colorado, and find the closest stores. This is a standard
feature on many company websites with physical locations.</p>
<p>During my investigation, I discovered that the SQLite database has an extension
called Spatialite, which can perform many tasks that the PostGIS extension can
do. If you’re unfamiliar with SQLite, it differs from a standard database in
that it doesn’t rely on a client-server protocol but works with a local file.
Since this data won’t change frequently, this is a suitable solution. However, I
encountered some technical challenges when trying to use the extension.</p>
<p>In Go, the most popular library for SQLite is
<a href="https://github.com/mattn/go-sqlite3" rel="nofollow">go-sqlite3</a>. It requires CGO, which may
deter some programmers, but I didn’t have a hard requirement for or against CGO
– I just needed to build a working feature.</p>
<p>The library allows extensions to be loaded at runtime. These extensions are
named after their compiled dynamic library. For the latest version of
Spatialite, the code looks like this:</p>
<pre><code><span><span><span>sql</span><span>.</span><span>Register</span><span>(</span><span>&#34;sqlite_with_spacialite&#34;</span><span>,</span> <span>&amp;</span><span>sqlite</span><span>.</span><span>SQLiteDriver</span><span>{</span>
</span></span><span><span>	<span>Extensions</span><span>:</span> <span>[]</span><span>string</span><span>{</span>
</span></span><span><span>		<span>&#34;mod_spatialite&#34;</span><span>,</span>
</span></span><span><span>	<span>},</span>
</span></span><span><span><span>})</span>
</span></span></code></pre><p>This is then used with Golang’s <code>database/sql</code> library with the custom driver
name.</p>
<pre><code><span><span><span>db</span><span>,</span> <span>err</span> <span>:=</span> <span>sql</span><span>.</span><span>Open</span><span>(</span><span>&#34;sqlite_with_spacialite&#34;</span><span>,</span> <span>&#34;locations.sqlite&#34;</span><span>)</span>
</span></span><span><span><span>if</span> <span>err</span> <span>!=</span> <span>nil</span> <span>{</span>
</span></span><span><span>	<span>return</span> <span>nil</span><span>,</span> <span>fmt</span><span>.</span><span>Errorf</span><span>(</span><span>&#34;could not open database file: %w&#34;</span><span>,</span> <span>err</span><span>)</span>
</span></span><span><span><span>}</span>
</span></span></code></pre><p>This is a simple setup. However, I ran into issues because of being on a Mac
with Apple Silicon. This is where this blog post transitions into debugging
mode.</p>
<p>I installed SQLite and Spatialite using <a href="https://brew.sh/" rel="nofollow">brew</a>.</p>
<pre><code><span><span>brew install sqlite3 libspatialite spatialite-tools
</span></span></code></pre><h2 id="debugging-step-by-step"><a href="#debugging-step-by-step" rel="nofollow">#</a> Debugging step by step</h2>
<p>So, let’s dive in and explore how I successfully debugged and resolved the
problem with the help of Google, GitHub Issues, and a bit of persistence!</p>
<p>When I ran the code, it would not execute due to an error related to the dynamic
module compiled for Spatialite.</p>
<pre><code>dlopen(mod_spatialite.dylib, 0x000A): tried: &#39;mod_spatialite.dylib&#39; (no such file), &#39;/System/Volumes/Preboot/Cryptexes/OSmod_spatialite.dylib&#39; (no such file), &#39;/usr/lib/mod_spatialite.dylib&#39; (no such file, not in dyld cache), &#39;mod_spatialite.dylib&#39; (no such file), &#39;/usr/local/lib/mod_spatialite.dylib&#39;
</code></pre>
<p>This error message indicates that the dynamic module could not be found in a
lookup path. I’ve enough experience with compilation to understand that. I had
to spend some time researching:</p>
<ul>
<li>Where is the <code>mod_spatialite.dylib</code> located?</li>
<li>How to tell CGO where to look up that file.</li>
</ul>
<blockquote>
<p>Google search: <code>brew mod_spatialite.dylib</code>.</p>
</blockquote>
<p>On Apple Silicon, <code>brew</code> changed the location of the installation directory to
<code>/opt/homebrew</code> (see this
<a href="https://github.com/Homebrew/brew/issues/14810" rel="nofollow">issue</a>). I found it at
<code>/opt/homebrew/lib/mod_spatialite.dylib</code>.</p>
<p>With CGO, there are options to change the lookup path for libraries via
<code>ldflags</code>. The <a href="https://pkg.go.dev/cmd/link" rel="nofollow">docs</a> state that you can use <code>-r</code>
to set the dynamic linker search path.</p>
<pre><code><span><span>$ go run --ldflags<span>=</span>-r<span>=</span>/opt/homebrew/lib main.go
</span></span><span><span>fatal error: unexpected signal during runtime execution
</span></span><span><span><span>[</span>signal SIGSEGV: segmentation violation <span>code</span><span>=</span>0x2 <span>addr</span><span>=</span>0xc00000000 <span>pc</span><span>=</span>0x1028cc274<span>]</span>
</span></span></code></pre><p>This led to a runtime panic.</p>
<p>This means that the code compiled without a runtime error, but the panic caused
memory access issues. It took time to pinpoint the problem. I had some leads,
but I needed more concrete.</p>
<blockquote>
<p>Google search: <code>golang sqlite3 &#34;unexpected signal during runtime execution&#34;</code>.</p>
</blockquote>
<p>I searched through any GitHub issue that mentioned Go, SQLite, and SIGSEGV. This
<a href="https://github.com/canonical/dqlite/issues/179" rel="nofollow">issue</a> said a specific build
tag that could be used to compile the local install SQLite –
<code>--tags libsqlite3</code>.</p>
<pre><code><span><span>$ go run --ldflags<span>=</span>-r<span>=</span>/opt/homebrew/lib --tags libsqlite3 main.go
</span></span><span><span>../../go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3_load_extension.go:25:8: could not determine kind of name <span>for</span> C.sqlite3_enable_load_extension
</span></span><span><span>../../go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3_load_extension.go:78:8: could not determine kind of name <span>for</span> C.sqlite3_load_extension
</span></span></code></pre><p>The compilation is now failing, with no more runtime errors.</p>
<blockquote>
<p>Google search: <code>golang &#34;could not determine kind of name for&#34;</code>.</p>
</blockquote>
<p>Searches suggested that this error has to do with the compilation of the C code.
The recommendations were to remove a newline before an <code>import</code> call. In Go,
when importing C headers, the Go parser requires <code>import &#34;C&#34;</code> to be immediately
after them.</p>
<p>I didn’t have any <code>import &#34;C&#34;</code> in my codebase. After inspecting the file
<code>sqlite3_load_extension.go</code>, I didn’t see the problem mentioned. This required
some further searching.</p>
<blockquote>
<p>Google search: <code>golang &#34;could not determine kind of name for&#34; sqlite3</code>.</p>
</blockquote>
<p>Eventually I found a Github
<a href="https://github.com/mattn/go-sqlite3/issues/839#issuecomment-674074775" rel="nofollow">comment</a>,
which indicated that I needed to point to the <code>sqlite</code> headers. This required
setting <code>CGO_CFLAGS</code>. I had to find where the headers were for the <code>sqlite</code>
installation I had.</p>
<pre><code><span><span><span>`</span>$ find /opt/homebrew <span>|</span> grep sqlite <span>|</span> grep <span>&#39;\.h$&#39;</span>
</span></span><span><span>/opt/homebrew/Cellar/libspatialite/5.0.1_2/include/spatialite/sqlite.h
</span></span><span><span>/opt/homebrew/Cellar/sqlite/3.41.2/include/sqlite3.h
</span></span><span><span>/opt/homebrew/Cellar/sqlite/3.41.2/include/sqlite3ext.h
</span></span></code></pre><p>It looks like they’re located in <code>/opt/homebrew/Cellar/sqlite/3.41.2</code>, which
corresponds to the version that was installed.</p>
<pre><code><span><span>$ <span>CGO_CFLAGS</span><span>=</span><span>&#34;-I/opt/homebrew/Cellar/sqlite/3.41.2/include&#34;</span> <span>\
</span></span></span><span><span><span></span>  go run <span>\
</span></span></span><span><span><span></span>    --ldflags<span>=</span>-r<span>=</span>/opt/homebrew/lib <span>\
</span></span></span><span><span><span></span>    --tags libsqlite3 <span>\
</span></span></span><span><span><span></span>    main.go
</span></span><span><span>/opt/homebrew/Cellar/go/1.20.2/libexec/pkg/tool/darwin_arm64/link: running clang failed: <span>exit</span> status <span>1</span>
</span></span><span><span>ld: warning: directory not found <span>for</span> option <span>&#39;-L/usr/local/opt/sqlite/lib&#39;</span>
</span></span><span><span>Undefined symbols <span>for</span> architecture arm64:
</span></span><span><span>  <span>&#34;_sqlite3_enable_load_extension&#34;</span>, referenced from:
</span></span><span><span>      __cgo_621878b5dd3b_Cfunc_sqlite3_enable_load_extension in 000007.o
</span></span><span><span>     <span>(</span>maybe you meant: __cgo_621878b5dd3b_Cfunc_sqlite3_enable_load_extension<span>)</span>
</span></span><span><span>  <span>&#34;_sqlite3_load_extension&#34;</span>, referenced from:
</span></span><span><span>      __cgo_621878b5dd3b_Cfunc_sqlite3_load_extension in 000007.o
</span></span><span><span>     <span>(</span>maybe you meant: __cgo_621878b5dd3b_Cfunc_sqlite3_load_extension<span>)</span>
</span></span><span><span>ld: symbol<span>(</span>s<span>)</span> not found <span>for</span> architecture arm64
</span></span><span><span>clang: error: linker <span>command</span> failed with <span>exit</span> code <span>1</span> <span>(</span>use -v to see invocation<span>)</span>
</span></span></code></pre><p>Another compilation error. This time, however, it provided a clue to the issue.</p>
<p>The linker is searching for the SQLite dynamic library in the incorrect
location. I needed to find where <code>sqlite</code> had been compiled within
<code>/opt/homebrew</code> as a dynamic library.</p>
<pre><code><span><span>$ find /opt/homebrew <span>|</span> grep sqlite <span>|</span> grep dylib
</span></span><span><span>/opt/homebrew/Cellar/sqlite/3.41.2/lib/libsqlite3.0.dylib
</span></span><span><span>/opt/homebrew/Cellar/sqlite/3.41.2/lib/libsqlite3.dylib
</span></span></code></pre><p>To set the location of the dynamic library, you can use <code>CGO_LDFLAGS</code>.</p>
<pre><code><span><span><span>CGO_CFLAGS</span><span>=</span><span>&#34;-I/opt/homebrew/Cellar/sqlite/3.41.2/include&#34;</span> <span>\
</span></span></span><span><span><span></span><span>CGO_LDFLAGS</span><span>=</span><span>&#34;-L/opt/homebrew/Cellar/sqlite/3.41.2/lib&#34;</span> <span>\
</span></span></span><span><span><span></span>  go run <span>\
</span></span></span><span><span><span></span>    --ldflags<span>=</span>-r<span>=</span>/opt/homebrew/lib <span>\
</span></span></span><span><span><span></span>    --tags libsqlite3 <span>\
</span></span></span><span><span><span></span>    main.go
</span></span><span><span><span># command-line-arguments</span>
</span></span><span><span>ld: warning: directory not found <span>for</span> option <span>&#39;-L/usr/local/opt/sqlite/lib&#39;</span>
</span></span><span><span>INFO program started
</span></span></code></pre><p>It’s working! There’s still a warning message, but I’m not overly concerned.
Warnings can be treated as recommendations.</p>
<p>I’ve set up everything above in environment variables within a <code>.envrc</code> file.
This allows me to avoid polluting the commands and not have to remember them.</p>
<pre><code><span><span><span>export</span> <span>GOFLAGS</span><span>=</span><span>&#34;--tags=libsqlite3 --ldflags=-r=/opt/homebrew/lib&#34;</span>
</span></span><span><span><span>export</span> <span>CGO_LDFLAGS</span><span>=</span><span>&#34;-L/opt/homebrew/Cellar/sqlite/3.41.2/lib&#34;</span>
</span></span><span><span><span>export</span> <span>CGO_CFLAGS</span><span>=</span><span>&#34;-I/opt/homebrew/Cellar/sqlite/3.41.2/include&#34;</span>
</span></span></code></pre><p>This solution works, but there needs to be an issue with it. If brew upgrades
sqlite, I’ll need to update the directories. This can be fixed with some
directory lookups within the .envrc script, but at this point, I’m done
investigating this.</p>
<p>By leveraging Google and GitHub Issues, I could identify the root cause of the
problem, explore various solutions, and ultimately find the one that worked for
my situation. As developers, we often encounter challenges that require us to
dig deep and think critically. I hope this post has shown you the value of
perseverance and resourcefulness when tackling such issues. Happy coding!</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-03-29.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Steps toward automation</title>
      <link>https://jtarchie.com/posts/2023-03-21.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="steps-toward-automation"><a href="#steps-toward-automation" rel="nofollow">#</a> Steps toward automation</h1>
<p>In this blog post, I’ll be discussing my approach to automating tasks that
software engineers have to perform for their job. Before diving into the core
concept, I’d like to define and narrow down the idea of automation within
software engineering.</p>
<p>Automation, in this context, refers to code created by programmers to simplify
tasks. The users benefiting from this automation are the computer programmers
themselves. The platforms on which programmers automate tasks include
development machines, CI/CD systems, cron jobs, and more. Typically, there is a
trigger for the automated task, such as an updated file, a new version of a
container, or the presence of a binary file. The job may involve security
scanning, updating downstream dependencies, or sending Slack messages.</p>
<p>Automation complexity arises when scripts have interdependencies, often
represented as a pipeline in a CI/CD system like GitHub Actions, GoCD, or
Concourse. An example pipeline can be a new git commit that happened in the code
base, tests need to run against the code, and when they pass, the code gets
deployed into an environment, which has performance and end-to-end tests to make
sure there was no regression. If anything fails in the pipeline, you can rerun
any part of the pipeline just in case.</p>
<p>As a software engineer, I’ve established two ground rules around automation.</p>
<p><img src="/assets/2023-03-21.jpg" alt="Robots are often seen as automation. We are not robots, however"/></p>
<p>First, “Automate yourself out of a job,” a mindset more than something precisely
actionable. If you find yourself repeatedly performing a task that doesn’t
require critical thinking but still consumes time, try to figure out how to
automate it. For example, if you’re always responsible for deployments and it
takes up a significant portion of your time, automate the process to free up
your schedule.</p>
<p>My second rule for automation is, “Only to automate things you can do manually.”
Many engineers, myself included, have fallen into the trap of automating a task
before fully understanding the problem, leading to spectacular failures when
there’s no manual fallback. To avoid this, ensure you know the commands and
requirements for manually executing the automated task in emergencies.</p>
<p>Applying these two rules will help you identify the repetitive tasks you’re
performing and automate them with confidence. Enabling you to share your
understanding with team members and provide context.</p>
<p>I once worked on an application where our platform was a CI/CD system that
allowed people to install and update a platform. We provided bundled scripts,
which meant most users never saw the commands contained within the scripts. We
embedded these scripts and commands in our documentation to maintain
transparency, allowing users to run them from their development environment.</p>
<p>We wanted to prevent users from encountering issues when copying and pasting
commands from the documentation, such as syntax errors or incorrect arguments.
Applying the two rules ensured the commands worked manually and within our CI/CD
system. First, we manually tested the commands before adding them to the
documentation. Second, we used the bundled scripts in our CI/CD system to
confirm they also worked there. This approach led to documentation with its own
CI/CD, ensuring everything was valid and functional. It was an advantageous
experience.</p>
<p>With any rules, there are always going to be edge cases. For example, there may
be security constraints with running specific code. The condition requires that
code be executed within a particular environment because the credentials and
network access have been isolated. You’ll never be able to run this locally from
your development machine.</p>
<p>Also, with manual and automated intervention, there will be times when you want
to do something manually constantly. It could be for the satisfaction,
validation, and verification that it Just Works™ correctly. It could also be
that you are the only one that runs this. Where the line exists between
automation and manual execution is a personal/team preference; just make sure to
check yourself that your automation is not over-engineering a straightforward
process.</p>
<p>In conclusion, automation in software engineering can be a powerful tool to save
time, reduce human error, and streamline processes. By adhering to the two
ground rules of automation and considering edge cases and personal/team
preferences, you can create a robust automation system that benefits you and
your team. Remember, automation aims to make life easier for you and your fellow
engineers.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-03-21.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Programming Language over Data language</title>
      <link>https://jtarchie.com/posts/2023-03-05.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="programming-language-over-data-language"><a href="#programming-language-over-data-language" rel="nofollow">#</a> Programming Language over Data language</h1>
<p>When it comes to configuration languages, the end goal is usually to use a
language that provides human-readable data structures. The most modern language
for this purpose is YAML, from a syntax point of view. It’s easy to work with in
terms of data type structure and readability. However, when it comes to
templating, those features are not enough. Usually, composition, inheritance,
type information, and validation of data structures are needed to.</p>
<p>After choosing YAML, most people have to come up with their own tooling and
patterns to create templates. The original motivation for using templates is to
facilitate shared configuration. This is not just a matter of a single Boolean
value for scaling the number of instances or a simple true/false statement for
enabling/disabling something. It often requires structures with multiple levels
and shapes.</p>
<p>To illustrate the complexity of configuration, let’s look at an example from
Kubernetes.</p>
<pre><code><span><span><span>apiVersion</span><span>:</span><span> </span><span>v1</span><span>
</span></span></span><span><span><span></span><span>kind</span><span>:</span><span> </span><span>Pod</span><span>
</span></span></span><span><span><span></span><span>metadata</span><span>:</span><span>
</span></span></span><span><span><span>  </span><span>name</span><span>:</span><span> </span><span>mypod</span><span>
</span></span></span><span><span><span></span><span>spec</span><span>:</span><span>
</span></span></span><span><span><span>  </span><span>containers</span><span>:</span><span>
</span></span></span><span><span><span>    </span>- <span>name</span><span>:</span><span> </span><span>mycontainer</span><span>
</span></span></span><span><span><span>      </span><span>image</span><span>:</span><span> </span><span>myimage</span><span>
</span></span></span><span><span><span>      </span><span>resources</span><span>:</span><span>
</span></span></span><span><span><span>        </span><span>limits</span><span>:</span><span>
</span></span></span><span><span><span>          </span><span>memory</span><span>:</span><span> </span><span>&#34;64Mi&#34;</span><span>
</span></span></span><span><span><span>          </span><span>cpu</span><span>:</span><span> </span><span>&#34;500m&#34;</span><span>
</span></span></span><span><span><span>        </span><span>requests</span><span>:</span><span>
</span></span></span><span><span><span>          </span><span>memory</span><span>:</span><span> </span><span>&#34;32Mi&#34;</span><span>
</span></span></span><span><span><span>          </span><span>cpu</span><span>:</span><span> </span><span>&#34;250m&#34;</span><span>
</span></span></span><span><span><span>    </span>- <span>name</span><span>:</span><span> </span><span>anothercontainer</span><span>
</span></span></span><span><span><span>      </span><span>image</span><span>:</span><span> </span><span>ubuntu</span><span>
</span></span></span><span><span><span>      </span><span>resources</span><span>:</span><span>
</span></span></span><span><span><span>        </span><span>limits</span><span>:</span><span>
</span></span></span><span><span><span>          </span><span>memory</span><span>:</span><span> </span><span>&#34;32Mi&#34;</span><span>
</span></span></span></code></pre><p>We can see that the <code>containers</code> can be a list, with each entry having
<code>resources</code>, with multiple nobs and dials. There are user experience issues with
the above:</p>
<ul>
<li>Are the values for <code>resources</code> required or optional?</li>
</ul>
<p>Via the <code>kubectl</code> CLI, we can ask the server for verification the values are
correct.</p>
<ul>
<li>
<p>Can we set the same value for <code>memory</code> in all properties?</p>
</li>
<li>
<p>Can we set all the <code>containers</code> to the same properties, but just change one
for <code>mycontainer</code>?</p>
<p>With YAML anchors, and :</p>
<p>We can use them for a value:</p>
<pre><code><span><span><span>memory</span><span>:</span><span> </span><span>&amp;memory</span><span> </span><span>&#34;32Mi&#34;</span><span>
</span></span></span><span><span><span></span><span>another_memory</span><span>:</span><span> </span><span>*memory</span><span>
</span></span></span></code></pre><p>Ability to override the value :</p>
<pre><code><span><span><span>limits</span><span>:</span><span> </span><span>&amp;limits</span><span>
</span></span></span><span><span><span>  </span><span>memory</span><span>:</span><span> </span><span>&#34;32Mi&#34;</span><span>
</span></span></span><span><span><span>  </span><span>cpu</span><span>:</span><span> </span><span>&#34;500m&#34;</span><span>
</span></span></span><span><span><span></span><span># ... some lines far away ...</span><span>
</span></span></span><span><span><span></span><span>limits</span><span>:</span><span>
</span></span></span><span><span><span>  </span><span>&lt;&lt;</span><span>:</span><span> </span><span>*limits</span><span>
</span></span></span><span><span><span>  </span><span>memory</span><span>:</span><span> </span><span>&#34;64Mi&#34;</span><span>
</span></span></span></code></pre></li>
<li>
<p>Can we use <code>resources</code> to all <code>containers</code> that are configured for this
cluster? Not just this file.</p>
</li>
<li>
<p>Can we use this is a base template and received semantic updates without
interference?</p>
</li>
<li>
<p>Can we use conditionals for parts of the configuration?</p>
<p>Yes and no. There is tooling that has been created to do this, such as <code>helm</code>
charts. It is industry standard and clunky.</p>
</li>
</ul>
<p>As a result, the pattern of needing to come up with these templates using YAML
just keeps happening. We need to accept alternatives such as Lua, Python, or
JavaScript/Typescript, which have the required features to help us.</p>
<p>To demonstrate how Typescript can be used to rewrite the previous configuration,
let’s take a look at the following code:</p>
<pre><code><span><span><span>const</span> <span>pod</span> <span>=</span> <span>{</span>
</span></span><span><span>  <span>apiVersion</span><span>:</span> <span>&#34;v1&#34;</span><span>,</span>
</span></span><span><span>  <span>kind</span><span>:</span> <span>&#34;Pod&#34;</span><span>,</span>
</span></span><span><span>  <span>metadata</span><span>:</span> <span>{</span>
</span></span><span><span>    <span>name</span><span>:</span> <span>&#34;mypod&#34;</span><span>,</span>
</span></span><span><span>  <span>},</span>
</span></span><span><span>  <span>spec</span><span>:</span> <span>{</span>
</span></span><span><span>    <span>containers</span><span>:</span> <span>[</span>
</span></span><span><span>      <span>{</span>
</span></span><span><span>        <span>name</span><span>:</span> <span>&#34;mycontainer&#34;</span><span>,</span>
</span></span><span><span>        <span>image</span><span>:</span> <span>&#34;myimage&#34;</span><span>,</span>
</span></span><span><span>        <span>resources</span><span>:</span> <span>{</span>
</span></span><span><span>          <span>limits</span><span>:</span> <span>{</span>
</span></span><span><span>            <span>memory</span><span>:</span> <span>&#34;64Mi&#34;</span><span>,</span>
</span></span><span><span>            <span>cpu</span><span>:</span> <span>&#34;500m&#34;</span><span>,</span>
</span></span><span><span>          <span>},</span>
</span></span><span><span>          <span>requests</span><span>:</span> <span>{</span>
</span></span><span><span>            <span>memory</span><span>:</span> <span>&#34;32Mi&#34;</span><span>,</span>
</span></span><span><span>            <span>cpu</span><span>:</span> <span>&#34;250m&#34;</span><span>,</span>
</span></span><span><span>          <span>},</span>
</span></span><span><span>        <span>},</span>
</span></span><span><span>      <span>},</span>
</span></span><span><span>      <span>{</span>
</span></span><span><span>        <span>name</span><span>:</span> <span>&#34;anothercontainer&#34;</span><span>,</span>
</span></span><span><span>        <span>image</span><span>:</span> <span>&#34;ubuntu&#34;</span><span>,</span>
</span></span><span><span>        <span>resources</span><span>:</span> <span>{</span>
</span></span><span><span>          <span>limits</span><span>:</span> <span>{</span>
</span></span><span><span>            <span>memory</span><span>:</span> <span>&#34;32Mi&#34;</span><span>,</span>
</span></span><span><span>          <span>},</span>
</span></span><span><span>        <span>},</span>
</span></span><span><span>      <span>},</span>
</span></span><span><span>    <span>],</span>
</span></span><span><span>  <span>},</span>
</span></span><span><span><span>};</span>
</span></span></code></pre><p>In this TypeScript code, we have defined a type for <code>Pod</code> and <code>Container</code> to
ensure that the values provided are optional, required, and typed correctly. By
doing so, we get the benefits of type safety and can catch errors early in the
development process.</p>
<pre><code><span><span><span>interface</span> <span>Pod</span> <span>{</span>
</span></span><span><span>  <span>apiVersion</span>: <span>string</span><span>;</span>
</span></span><span><span>  <span>kind</span>: <span>string</span><span>;</span>
</span></span><span><span>  <span>metadata</span><span>:</span> <span>{</span>
</span></span><span><span>    <span>name</span>: <span>string</span><span>;</span>
</span></span><span><span>  <span>};</span>
</span></span><span><span>  <span>spec</span><span>:</span> <span>{</span>
</span></span><span><span>    <span>containers</span>: <span>Container</span><span>[];</span>
</span></span><span><span>  <span>};</span>
</span></span><span><span><span>}</span>
</span></span><span><span>
</span></span><span><span><span>interface</span> <span>Container</span> <span>{</span>
</span></span><span><span>  <span>name</span>: <span>string</span><span>;</span>
</span></span><span><span>  <span>image</span>: <span>string</span><span>;</span>
</span></span><span><span>  <span>resources</span><span>:</span> <span>{</span>
</span></span><span><span>    <span>limits</span><span>:</span> <span>{</span>
</span></span><span><span>      <span>memory</span>: <span>string</span><span>;</span>
</span></span><span><span>      <span>cpu?</span>: <span>string</span><span>;</span>
</span></span><span><span>    <span>};</span>
</span></span><span><span>    <span>requests</span><span>?:</span> <span>{</span>
</span></span><span><span>      <span>memory</span>: <span>string</span><span>;</span>
</span></span><span><span>      <span>cpu?</span>: <span>string</span><span>;</span>
</span></span><span><span>    <span>};</span>
</span></span><span><span>  <span>};</span>
</span></span><span><span><span>}</span>
</span></span></code></pre><p>Additionally, we could use <code>npm</code> to inherit a versioned definition of the <code>Pod</code>
and <code>Container</code> types and override what we need to. This allows us to reuse
common configuration patterns across projects and maintain versioning control.
There could be a type for <code>Pod</code> that is always configured correctly for the
<code>apiVersion</code>.</p>
<p>Using TypeScript for configuration provides type discoverability, allowing for
quick and easy identification of available fields and expected types. This
feature is especially helpful when working in an editor like Visual Studio Code,
which provides IntelliSense for writing configuration files. With IntelliSense,
you can get suggestions and auto-completion for configuration fields, providing
a pleasant experience for the programmer.</p>
<p>In addition to the benefits mentioned above, it’s also possible to output the
structure of a TypeScript configuration into a YAML file. While this adds an
extra layer of abstraction, it can be managed effectively for projects that
cannot change their default configuration language.</p>
<p>This isn’t an argument for the exclusive use of JavaScript/TypeScript, but
rather that it offers similar functionality to popular configuration languages
like YAML, <a href="https://jsonnet.org/" rel="nofollow">Jsonnet</a>, <a href="https://cuelang.org/" rel="nofollow">CUE</a>,
<a href="https://github.com/hashicorp/hcl" rel="nofollow">HCL</a>, <a href="https://carvel.dev/ytt/" rel="nofollow">YTT</a>,
<a href="https://toml.io/en/" rel="nofollow">Toml</a>, <a href="https://www.json.org/json-en.html" rel="nofollow">JSON</a>, and
more. While the actual configuration may not change significantly, using
JavaScript/TypeScript can add built-in functionality. As a personal statement, I
would caution against trying to create a configuration language, as it may end
up being a poorly implemented programming language.</p>
<p>YAML may lack the required features for complex configurations, leading to the
use of templates that can add complexity. TypeScript offers explicit typing,
versioning control, and the ability to reuse common configuration patterns,
making it a strong contender for configuration languages. By choosing the right
language and implementing best practices, you can ensure your configuration is
well-structured, maintainable, and scalable for your application’s success.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2023-03-05.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Testing Thoughts</title>
      <link>https://jtarchie.com/posts/2022-02-17.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="testing-thoughts"><a href="#testing-thoughts" rel="nofollow">#</a> Testing Thoughts</h1>
<p>When it comes to testing, the simplest explanation I have for tests is:</p>
<blockquote>
<p>Users use our code through the web, console, and other user interfaces. Tests
are another interface just for the developers.</p>
</blockquote>
<p>Tests provide the developers confidence that things are working the way they
should.</p>
<h2 id="what-does-it-mean-to-be-red-then-green"><a href="#what-does-it-mean-to-be-red-then-green" rel="nofollow">#</a> What does it mean to be red then green?</h2>
<p>I always remember “red, green, then refactor” when I write a test.</p>
<ol>
<li>
<p>Red</p>
<p>An assertion (in the test) was made on the code’s behavior. It is failing on
an assertion test.</p>
</li>
<li>
<p>Green</p>
<p>Write the code to make that tests pass. Now you have something in a working
state. Commit it!</p>
</li>
<li>
<p>Refactor</p>
<p>Now that you know things are in a working and deliverable state, can you
refactor your code?</p>
</li>
</ol>
<p>Do you need to do this each time? No.</p>
<p>It helps me narrow down the scope of a test and have a criteria for when I am
<em>done</em> with it.</p>
<h2 id="is-your-test-testing-just-your-test-or-your-code"><a href="#is-your-test-testing-just-your-test-or-your-code" rel="nofollow">#</a> Is your test testing just your test or your code?</h2>
<p>Imagine this a situation while programming.</p>
<p>There is a test, some code was written, and everything is green. Someone comes
back, edits some code, all the tests are still green.</p>
<p>This is great! Tests should have to change with a different implementation of
code, just as long as the assertions of the behavior is the same.</p>
<p>When you comment out your code, your test should fail. Yet it is still passing.
Why is that?</p>
<p>Sometimes our tests are just testing our tests, not our code. Ee might be
testing a mock implementation of something.</p>
<p>Tests should fail when I comment out the code I believe to be testings. It may
not be the tests I wrote, but something should break, so we know something
changed.</p>
<h2 id="how-do-you-start-writing-tests"><a href="#how-do-you-start-writing-tests" rel="nofollow">#</a> How do you start writing tests?</h2>
<p>When writing tests, I like to think of the happy path first, “If everything was
working perfectly, what would I expect to happen?” Allowing you to write tests
free from the distractions of errors and side effects. It can help with test
setup the behavior you want the tests.</p>
<p>The next step is a sad path – errors, side effects, and other paths. For
example, “If we cannot connect to the API, what should we do?” It helps test the
<em>branching</em> (conditions) of your code.</p>
<p>Thinking about your tests in discrete chunks will help produce understandable
tests. Tests are another means to document your code, similar to commenting.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2022-02-17.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>When in San Diego</title>
      <link>https://jtarchie.com/posts/2022-01-03.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="when-in-san-diego"><a href="#when-in-san-diego" rel="nofollow">#</a> When in San Diego</h1>
<p>I’ve had some friends move to San Diego. Since I used to live there, this is a
list of things to do and try.</p>
<ul>
<li>🛣Coastal Highway 101, drive it from Carlsbad down to San Diego.</li>
<li>Drive to Torrey Pines, both South and North. It is just beautiful.</li>
</ul>
<h2 id="downtown"><a href="#downtown" rel="nofollow">#</a> Downtown</h2>
<ul>
<li>🍩<a href="https://goo.gl/maps/TvzidGvVLwUQhqZP7" rel="nofollow">Donut Bar</a> is excellent. I love
donuts and this place has never done wrong in providing something unique.</li>
<li>🥮<a href="https://goo.gl/maps/Am2HRpEPFyjPaRDMA" rel="nofollow">Extraordinary Desserts</a> has a menu
of just desserts. You may see a pattern here.</li>
<li>🌻<a href="https://goo.gl/maps/MA8FzMjvaGviTr3E7" rel="nofollow">Balboa Park</a> is a beautiful
botanical park. You can spend a day here, between the museums and gardens.</li>
<li>🦓<a href="https://goo.gl/maps/nGLYh58XAgr1vEz49" rel="nofollow">San Diego Zoo</a> is the world-famous
Zoo. Huge! All the animals! And located in Balboa Park.</li>
</ul>
<h2 id="breweries-"><a href="#breweries-" rel="nofollow">#</a> Breweries 🍺🍻</h2>
<p>This deserved its own section. These breweries are spread amongst San Diego.
Always ask the brewer/tender where to go next, they know more.</p>
<ul>
<li><a href="https://goo.gl/maps/7XzSgLuQBasaRmXT8" rel="nofollow">Modern Times</a> great experimental
brewery. Always something clever on tap. Similar to Something Brewery in
Brighton, CO.</li>
<li>Stone runs <a href="https://goo.gl/maps/Nf6BUG6zVCkHzRsD8" rel="nofollow">Stone Gardens</a> Brewing.
They have great location downtown and some of the best IPAs ever.</li>
<li><a href="https://goo.gl/maps/heg97DFVNgmSAnPv6" rel="nofollow">Resident Brewing</a> has a great downtown
SD location.</li>
</ul>
<h2 id="pacific-beach-pb-"><a href="#pacific-beach-pb-" rel="nofollow">#</a> Pacific Beach (PB) 🏴‍☠️</h2>
<p>A surf “college” town, but lots of food, drinks, etc. Grand Ave is the “main
street”, but the beauty of PB is the beach.</p>
<p>Fact: You used to be able to drink on the beach (like with kegs), until 2007
where the fight between July 4th beach party-goers and Police broke out.</p>
<ul>
<li>🍔<a href="https://goo.gl/maps/MJQfYwVKPjtzzv8s6" rel="nofollow">Kono’s Cafe</a> is a “Hawaiian”
breakfast spot. It is delicious. A tried-and-true hangover recovery area. It
is also at the start of the PB boardwalk.</li>
<li>PB Boardwalk, rent a bike (or bring one) and ride up and down. There are
<a href="https://www.youtube.com/watch?v=Xn87-mcnoVc" rel="nofollow">characters</a> along here, but it
can be very entertaining.</li>
<li>🏄‍♀️<a href="https://goo.gl/maps/uxWHo1LG4rVdkQUS9" rel="nofollow">Mission Beach Surfing</a> is where I
learned to surf. Every one fun and friendly. The water will be cold, it is the
Pacific.</li>
<li>🖼<a href="https://goo.gl/maps/FHetyMe9iQRXRPmx6" rel="nofollow">Green Flash Art</a> is a piece made to
represent the sunset. The lore of it, when looking at the sunset, you see a
green flash amongst the orange and red at the very end of it.</li>
<li><a href="https://g.page/figtreecafe-pacific-beach?share" rel="nofollow">Fig Tree Cafe</a> is a classic
hipster brunch – french toast, omelets, etc. Nothing will be deconstructed,
though.</li>
</ul>
<h2 id="solana-beach"><a href="#solana-beach" rel="nofollow">#</a> Solana Beach</h2>
<p>Standard beach-side town. Easily drive-through but a good place to stop.</p>
<ul>
<li>🍕&amp;🍺<a href="https://goo.gl/maps/HxM7Pb6ykJCVS7rP9" rel="nofollow">Pizza Port</a> is a brewery and
pizzeria. The best (maybe just nostalgic). My go-to here was the Shark Bite
Red Ale with BBQ Chicken, bacon, and pineapple (don’t judge) pizza.</li>
<li><a href="https://goo.gl/maps/sPAGwVtcoskKxLv38" rel="nofollow">Chart House</a> for the best seafood. It
is a swanky place, but worth it. They have a sister place across the street,
but I don’t see it anymore.</li>
</ul>
<h2 id="encinitas"><a href="#encinitas" rel="nofollow">#</a> Encinitas</h2>
<p>This is a classic surf-side town. Tony Hawk is from here!</p>
<ul>
<li>👓<a href="https://goo.gl/maps/95hpuTou61oHanRZA" rel="nofollow">Swamis</a> is a surf beach. If you are
here on a big wave day, you will see some of the best surfers in the world. Go
early in the morning, grab a cup of coffee, and enjoy.</li>
<li>☕️<a href="https://goo.gl/maps/wKXv6DKMjewZFPtj7" rel="nofollow">Lofty Coffee</a> grab a coffee here,
take it to Swamis.</li>
<li>🌯<a href="https://goo.gl/maps/qjfzcQMDqy2tDEFcA" rel="nofollow">Filberto’s</a> for a burrito. It is
open 24/7, may not be the best, but always accessible.</li>
</ul>
<h2 id="del-mar"><a href="#del-mar" rel="nofollow">#</a> Del Mar</h2>
<p>Just park and walk around.</p>
<ul>
<li>🥪<a href="https://goo.gl/maps/CPRG135MeifmKCNz9" rel="nofollow">Board and Brew</a> is a sandwich shop.
Get the chicken club with the Sweet &amp; Sour dipping sauce. Ask for the sauce
during purchase, it is not on the menu!</li>
<li>🌅<a href="https://goo.gl/maps/9DvUJR158xnmUYKKA" rel="nofollow">Seagrove Park</a> watch the sunset here
one night. Just do it!</li>
<li>🌮<a href="https://goo.gl/maps/THKXwrf5fNkySaAa9" rel="nofollow">Roberto’s</a> is the taco place that
Blink 182 sang about. Order the fish and shrimp tacos with chips and guac. Sit
outside and enjoy the cove.</li>
</ul>
<h2 id="la-jolla"><a href="#la-jolla" rel="nofollow">#</a> La Jolla</h2>
<p>La Jolla is like Cherry Creek (with Boulder) on the beach. Expensive and
beautiful houses everywhere.</p>
<ul>
<li>🐋<a href="https://goo.gl/maps/qJ6MK2BTvv9FwWhq9" rel="nofollow">Seal Rock</a> you can view seals are
hanging out and sleeping on rocks.</li>
<li>🐡<a href="https://goo.gl/maps/aMMKYa7Wv7tY8WA9A" rel="nofollow">Birch Aquarium</a> is by far the best
aquarium I’ve been to. It is huge!</li>
</ul>
<h2 id="hillcrest-"><a href="#hillcrest-" rel="nofollow">#</a> Hillcrest 🏳️‍🌈</h2>
<p>This is the most excellent neighborhood downtown. Stay here. Coffee shops,
restaurants, etc. This place has grown so much, and I couldn’t tell you what’s
there anymore.</p>
<h2 id="old-town-"><a href="#old-town-" rel="nofollow">#</a> Old Town 🇲🇽</h2>
<p>Tourist trap of an <em>authenticate</em> Mexican town. I don’t recall this being worth
it.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2022-01-03.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Embracing Tech Debt as a Team</title>
      <link>https://jtarchie.com/posts/2021-08-06.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="embracing-tech-debt-as-a-team"><a href="#embracing-tech-debt-as-a-team" rel="nofollow">#</a> Embracing Tech Debt as a Team</h1>
<p>Working on a product team always has two primary goals: figuring out what you
are building and when you can deliver those features. Ideally, the team
maintains its delivery velocity indefinitely. As an engineer, you’re delivering,
but something starts to feel off. Features become harder to implement, and
there’s a refactoring you’ve been putting off. Eventually, you realize you can’t
keep delivering as you once did.</p>
<p>You’ve encountered the all-too-familiar challenge of technical debt.</p>
<p>It’s tempting to look for someone to blame:</p>
<ul>
<li>A former engineer who used to be on the team.</li>
<li>Overwhelming workloads and tight deadlines imposed by leaders.</li>
<li>The one that no one wants to hear: “The product manager won’t let me.”</li>
</ul>
<p>But pointing fingers won’t help. Technical debt is a symptom of a team
struggling to prioritize. Your team needs to create space for addressing its
needs. Some tasks are essential, while others can wait, and the key is to focus
on what’s truly important.</p>
<p>A product manager’s job is to say, “Let’s deliver features. How do we get
there?” It’s up to engineering to figure out the best way to achieve this.
Remember, the product manager is on your team to help accomplish the same goals.
Please include them in your mindset and process, and don’t place blame on one
person for any group setbacks. Work together to improve and resolve issues.</p>
<p>Tech debt can seem like an intimidating and undefined obstacle. We’ve accepted
that our products will never be “finished,” We’ll keep refining them through
continuous delivery. But too often, tech debt gets pushed into a “save it for
later” category. Instead, let’s change our perspective and remove the negative
connotation.</p>
<p>Think of code maintenance as gardening. It will grow beautiful if we
consistently prune, replant, and tend to our code. There’s no such thing as a
one-day fix for a gardener because it would be inefficient for the long-term
care of the garden.</p>
<p>Maintaining code should be a regular part of the development process, and you
can help your team overcome its tech debt by continuously keeping it in mind. So
let’s embrace it together and cultivate a thriving, collaborative environment
for everyone involved.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2021-08-06.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Exploring JIT execution of `luajit`</title>
      <link>https://jtarchie.com/posts/2020-02-11.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="exploring-jit-execution-of-luajit"><a href="#exploring-jit-execution-of-luajit" rel="nofollow">#</a> Exploring JIT execution of <code>luajit</code></h1>
<p>I’ve always been impressed with the <a href="http://luajit.org/" rel="nofollow">LuaJIT</a> runtime. For
years, Mike Pall has been praised (and rightfully so) for creating it. It is the
fastest runtime for a dynamic language.</p>
<p>I’ve taken an interest in experimenting with it.</p>
<p>It can optimize dynamic code to efficient assembly instructions.</p>
<p>For example, we are calculating the sum of 1 to 100.</p>
<pre><code><span><span><span>local</span> <span>sum</span> <span>=</span> <span>0</span>
</span></span><span><span><span>for</span> <span>i</span> <span>=</span> <span>1</span><span>,</span> <span>100</span> <span>do</span>
</span></span><span><span>	<span>sum</span> <span>=</span> <span>sum</span> <span>+</span> <span>i</span>
</span></span><span><span><span>end</span>
</span></span></code></pre><p>It generates efficient assembly. The assembly using <code>luajit -jdump</code>. This
outputs a lot, but our interest lies in the <code>LOOP</code> scope.</p>
<pre><code>-&gt;LOOP:
xorps xmm6, xmm6
cvtsi2sd xmm6, ebp
addsd xmm7, xmm6           # sum = sum + i
add ebp, +0x01             # i = i + 1
cmp ebp, +0x64             # if i &lt;= 100
jle -&gt;LOOP                 #   goto LOOP
jmp -&gt;EXIT
</code></pre>
<p>The numbers assigned to <code>sum</code> and <code>i</code> are used as typed information. These types
don’t change during the execution of the script, and this allows the JIT to make
efficient assembly instructions.</p>
<p>LuaJIT (like Lua) assumes all numbers are floating-point, though. The
instruction <code>cvtsi2sd</code> converts integers to floating-point. Though this may be
fast, it is an extraneous instruction.</p>
<p>I wanted to see if LuaJIT could just do integer-specific instructions by
providing hints to the JIT. Lua doesn’t have a static type system, though.</p>
<pre><code><span><span><span>local</span> <span>ffi</span> <span>=</span> <span>require</span><span>(</span><span>&#34;ffi&#34;</span><span>)</span>
</span></span><span><span><span>ffi.cdef</span> <span>[[
</span></span></span><span><span><span>    typedef struct { int v; } int_t;
</span></span></span><span><span><span>]]</span>
</span></span><span><span>
</span></span><span><span><span>local</span> <span>int</span> <span>=</span> <span>ffi.metatype</span><span>(</span><span>&#34;int_t&#34;</span><span>,</span> <span>{})</span>
</span></span><span><span><span>local</span> <span>sum</span> <span>=</span> <span>int</span><span>(</span><span>0</span><span>)</span>
</span></span><span><span><span>for</span> <span>i</span> <span>=</span> <span>1</span><span>,</span> <span>100</span> <span>do</span>
</span></span><span><span>	<span>sum.v</span> <span>=</span> <span>sum.v</span> <span>+</span> <span>i</span>
</span></span><span><span><span>end</span>
</span></span></code></pre><pre><code>-&gt;LOOP:
add ebp, ebx        # sum = sum + i
mov [rax+0x10], ebp # not sure what this is for
add ebx, +0x01      # i = i + 1
cmp ebx, +0x64      # if i &lt;= 100
jle -&gt;LOOP          #   goto LOOP
jmp -&gt;EXIT
</code></pre>
<p>We have some extraneous instructions, which don’t seem to be helpful here. Can
we get this loop to be tighter.</p>
<p>With type information from <code>ffi</code> struct forces the instruction integer addition
(<code>add</code>). The example above is simple and raw, but it shows how easy adding types
was. LuaJIT has ways of
<a href="http://luajit.org/ext_ffi_tutorial.html#metatype" rel="nofollow">abstracting the struct field</a>,
so the implementation does bleed through.</p>
<p>Using the metamethods, we can have the struct use the <code>+</code> operator.</p>
<pre><code><span><span><span>local</span> <span>ffi</span> <span>=</span> <span>require</span><span>(</span><span>&#34;ffi&#34;</span><span>)</span>
</span></span><span><span><span>ffi.cdef</span> <span>[[
</span></span></span><span><span><span>    typedef struct { int v; } int_t;
</span></span></span><span><span><span>]]</span>
</span></span><span><span>
</span></span><span><span><span>local</span> <span>int</span>
</span></span><span><span><span>int</span> <span>=</span> <span>ffi.metatype</span><span>(</span><span>&#34;int_t&#34;</span><span>,</span> <span>{</span>
</span></span><span><span>	<span>__add</span> <span>=</span> <span>function</span><span>(</span><span>a</span><span>,</span> <span>b</span><span>)</span> <span>return</span> <span>int</span><span>(</span><span>a.v</span> <span>+</span> <span>b</span><span>)</span> <span>end</span>
</span></span><span><span><span>})</span>
</span></span><span><span><span>local</span> <span>sum</span> <span>=</span> <span>int</span><span>(</span><span>0</span><span>)</span>
</span></span><span><span><span>for</span> <span>i</span> <span>=</span> <span>1</span><span>,</span> <span>100</span> <span>do</span>
</span></span><span><span>	<span>sum</span> <span>=</span> <span>sum</span> <span>+</span> <span>i</span>
</span></span><span><span><span>end</span>
</span></span></code></pre><pre><code>-&gt;LOOP: add ebp, ebx # sum = sum + i
add ebx, +0x01       # i = i + 1 cmp
ebx, +0x64           # if i &lt;= 100
jle -&gt;LOOP           # goto LOOP
jmp -&gt;EXIT
</code></pre>
<p>Here we’ve managed to take standard lua code and optimize it using the <code>ffi</code>
functionality. The JIT is using the type information for more optimized
assembly.</p>
<p>The loop cannot get any tighter unless we remove the loop entirely. I have not
been able to get LuaJIT to perform this optimization.</p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2020-02-11.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
    <item>
      <title>Hello, World!</title>
      <link>https://jtarchie.com/posts/2019-01-21.html</link>
      <description></description>
      <content:encoded><![CDATA[<h1 id="hello-world"><a href="#hello-world" rel="nofollow">#</a> Hello, World!</h1>
<p>I wrote my own blog engine. I wanted to. The other that exist are just fine.</p>
<p>When using it locally, it watches the content and rebuilds it. The browser still
requires a manual refresh, but that’s alright. It is developed using <code>ruby</code>.</p>
<p>It builds the site specifically for Github Pages. The <code>CNAME</code> file is customized
for this site.</p>
<p>All the posts are in the <code>posts</code> directory. They are all timestamped as
<code>YYYYMMDD.md</code>. The title used for navigation will always be single header <code>#</code>.</p>
<p>It can do fancy markdown things.</p>
<p>Source code rendering for <code>ruby</code>:</p>
<pre><code><span><span><span>puts</span> <span>&#34;Hello, World&#34;</span>
</span></span></code></pre><h2 id="smaller-header"><a href="#smaller-header" rel="nofollow">#</a> Smaller header</h2>
<p><del>don’t worry about it</del></p>
]]></content:encoded>
      <guid>https://jtarchie.com/posts/2019-01-21.html</guid>
      <pubDate>Thu, 29 Oct 2026 02:07:35 +0000</pubDate>
    </item>
  </channel>
</rss>