Phil Nash

5 quick tips for giving better presentations

Thu, 05 Mar 2026 00:00:00 GMT

I have been speaking publicly at developer conferences for over a decade and in that time I've seen plenty of other people giving talks. Everyone gives talks differently, and if you are a speaker or thinking about getting into it, I encourage you to work on developing your unique style as well as your content.

However, recently I have noticed a few little things that some speakers do and others do not that make for a better experience for both the audience and the presenter. They are mostly inconsequential for the content of a talk, but they can act as speed bumps that take the momentum out of a talk.

So, in no particular order, here are a few things you should remember when giving a talk.

Finish strong

Have you ever seen this situation play out?

A speaker wraps up their talk nicely with an insightful conclusion. Then they say something like, "Thank you very much. Any questions?"

The audience, prepared to bring the house down with cheering and clapping are now caught like a deer in headlights. Instead of rapturous applause an awkward silence descends. Someone puts their hand up and asks a question. Q&A ensues, but everything now feels a bit flat.

As an audience member, you want to show appreciation for a great job, so as a speaker you should give space for that. When concluding a talk, end with a "Thank you" or some other definitive way to indicate you are finished speaking, and let the crowd have their moment to thank you. Once the applause dies down, that's when you, or even better an MC or moderator, can indicate that it is time (or not) for questions.

Don't let a talk fizzle out into the Q&A portion, finish on a high. Deal with the questions afterwards.

Take your lanyard off

The conference lanyard is the sign that you are part of this temporary tribe that has formed. When wearing it, you can speak to anyone else also adorned in the conference colours knowing you have some common ground.

When you are on stage to deliver a talk, you no longer need the lanyard to signify you are part of the experience. But that's not why you should take it off. You should remove it for the duration of your talk simply because it gets in the way.

On stage there is more to consider. A lanyard hanging around your neck may get in the way of your laptop if you need to use it, it might bother the microphone cable, it might just not look that great. You probably didn't choose your outfit based on the conference colour scheme, so why throw a dangling colour clash into the mix?

Take your lanyard off for the talk, but don't forget to put it back on afterwards. You will want to help people remember who you are once you're no longer in the spotlight.

Keyboard shortcuts you wish you'd known for years

There are two macOS keyboard shortcuts that I think are vital to know if you want to make smooth transitions between parts of your talk.

Cmd + F1

Let's say you're presenting in Keynote and you're using the presenter view; you can see the notes on your laptop, the audience sees the slides on the screen. Your laptop is using the external screen as an extended display. Now you want to demo something. To do this you need to switch to mirroring, so that you can see the same demo on your laptop as the audience is seeing on the screen.

You could make this switch by opening the display settings and changing the setting. However, the keyboard shortcut is Cmd + F1. Flipping between mirroring and extending with a keypress is quicker, smoother and keeps the energy in your presentation. Cmd + F1, commit it to memory.

Cmd + Shift + F

If you're using Chrome and you want to go into fullscreen (with the green button in the top left of the window, or Shift + Fn + F) and you find that the address bar is still showing, taking up valuable screen real estate, that can be fixed. Cmd + Shift + F toggles whether the address bar shows. You're probably reading this in Chrome right now, try it out!

Resize the font, don't zoom the window

If you plan to live code or just show code within an IDE as part of a demo, you should make sure that the font is readable by the audience. I recommend doing this as part of a tech check before you are supposed to go on stage. This gives you the time to check things yourself and resize the text correctly.

You might think that zooming in, with the shortcut Cmd and + will help the audience see the code. In VS Code and other similar editors, that zooms the text and the entire interface. By the time the text is of a size that the audience will be able to read it, it will be crowded out by the file explorer and the terminal.

Instead, take the time to open the settings (Cmd + ,) and change the font size to something readable. That way the rest of the interface stays out of the way. If you plan to show things in the built-in terminal, make sure to increase the terminal font size too, it's a different setting.

Walk to the back of the room, or get a friend to check and give you a thumbs up, and make sure things are readable before you start the talk.

Trust in the tech

If you are in the fortunate position to be speaking at an event with a crew looking after you on stage, they will likely give you a microphone before you go on and be in control of it. Most of the time this means that the microphone is on and you needn't touch it, but it is muted at the sound desk. Then, when it is time for you to start they will unmute you and everyone will hear you.

It can be unnerving to believe that once you start speaking everyone will be able to hear you, but you should. Much like my first tip to finish strong, you also want to start strong, and "Hello, can you hear me? Is this on?" is not the way to achieve that.

Instead start by introducing yourself, start with a joke, start by thanking the audience for showing up. However you want to start, assume you will be heard and confidently start speaking. If something does go wrong, it's not your fault (unless you turned your microphone off on purpose). When things go right, you will capture the audience's attention and kick your session off in style.

Small tips, big impact

I have, of course, done the opposite of all of these things myself. I've started by asking if I can be heard, zoomed my editor until only the side panel was visible, fiddled in settings and menus to make my screen show the right thing, flapped my lanyard around on stage, and finished with "Thank you, any questions?" and a silent room. I only hope by sharing some of these tips that you can avoid all of those things yourself, sail through your talk with all your energy being used to wow the audience, and feel great at the end of it all.

So remember your shortcuts, take that lanyard off, trust that you will be heard, get your font sizes right, start with confidence and finish strong. I can't wait to see what you are going to talk about.

Things you need to do for npm trusted publishing to work

Sat, 31 Jan 2026 00:00:00 GMT

After the recent supply chain attacks on the npm ecosystem, notaby the Shai-Hulud 2.0 worm, GitHub took a number of actions to shore up the security of publishing packages to hopefully avoid further attacks. One of the outcomes was that long-lived npm tokens were revoked in favour of short-lived tokens or using trusted publishing.

I have GitHub Actions set up to publish new versions of npm packages that I maintain when a new tag is pushed to the repository. This workflow used long-lived npm tokens to authenticate, so when it came to updating a package recently I needed to update the publishing method too. The npm documentation on trusted publishing for npm packages was useful to a point, but there were some things I needed to do that the docs either didn't cover explicitly or weren't obvious enough, to get my package published successfully. I also came across this thread on GitHub where other people had similar issues. I wanted to share those things here.

TL;DR

Briefly, the changes that worked for me were to add the following to my GitHub Action publishing workflow:

# Permission to generate an OIDC token
permissions:
  id-token: write

jobs:
  publish:
    steps:
      ...
      # Ensure the latest npm is installed
      - run: npm install -g npm@latest
      ...
      # Add the --provenance flag to the publish command
      - run: npm publish --provenance

And ensure that the package.json refers to the correct repository:

{
  ...
  "repository": {
    "type": "git",
    "url": "git+https://github.com/${username}/${packageName}.git",
  },
  ...
}

For a bit more detail and alternative ways to set some of these settings, read on.

Package settings

Ok, so this is embarrassing, but initially I couldn't find the settings I needed to enable trusted publishing. The npm docs say:

Navigate to your package settings on npmjs.com and find the "Trusted Publisher" section.

I spent far too long looking around the https://www.npmjs.com/settings/${username}/packages page for the "Trusted Publisher" section. What I needed was the specific package settings, available here: https://www.npmjs.com/package/${packageName}/access.

You need to set up trusted publishing for each of your packages individually. That might be fine if you only maintain a few, it's going to be a huge hassle if you have a lot.

Once you have filled in the trusted publisher settings, then its on to updating your project so that it can be published successfully.

Permissions

This is in the npm docs, so I'm just including it for completeness. You need to give the workflow permission to generate an OIDC token that it can then use to publish the package. To do this requires one permission being set in your workflow file.

permissions:
  id-token: write

npm version

The docs clearly call out that:

Note: Trusted publishing requires npm CLI version 11.5.1 or later.

I needed to upgrade the version of npm used by my GitHub Actions workflow, so I added a simple step to install the latest version of npm as part of the run before publishing:

- run: npm install -g npm@latest

Automatic provenance

The docs also say:

When you publish using trusted publishing, npm automatically generates and publishes provenance attestations for your package. This happens by default—you don't need to add the --provenance flag to your publish command.

I did not find this to be the case. I needed to add the --provenance flag so that my package would publish successfully.

run: npm publish --provenance

This was something that seemed to help others too. You may only need to pass --provenance the first time, with it continuing to work automatically beyond that, but it can't hurt to keep it in your publish script (for when you need to update another package and you copy things over).

You can also set your package to generate provenance attestations on publishing by setting the provenance option in publishConfig in your package.json file.

{
  ...
  "publishConfig": {
    "provenance": true
  }
  ...
}

Or you can set the NPM_CONFIG_PROVENANCE environment variable.

env:
  NPM_CONFIG_PROVENANCE: true
run: npm publish

Repository details

Finally, I don't know if this last part helped as I did already have it set, but others in this GitHub thread found that setting the repository field in the package's package.json to specifically point to the GitHub repository also helped.

{
  ...
  "repository": {
    "type": "git",
    "url": "git+https://github.com/${username}/${packageName}.git",
  },
  ...
}

When you set up your trusted publisher in npm you do have to provide the repository details, so it makes sense to me that the package should agree with those details too.

Keep the ecosystem safe

Short-lived tokens, trusted publishing, and provenance all help keep the entire ecosystem safe. If you've read this far, it is because you are also updating your packages to publish with this method

I know there are people out there with many more packages, and packages that are much more popular than any of mine, but I hope this helps. It does amuse me that I went through this for a package that I'm pretty sure I'm the only user of, but at least I now know how to do it for the future.

I hope to see trusted publishing continue to expand to more providers, it is limited to GitHub and GitLab at the time of writing, and to be used by more packages. And I hope to see fewer worms charging through the package ecosystem and threatening all of our applications in the future.

How wrong can a JavaScript Date calculation go?

Sun, 11 Jan 2026 00:00:00 GMT

The Date object in JavaScript is frequently one that causes trouble. So much so, it is set to be replaced by Temporal soon. This is the story of an issue that I faced that will be much easier to handle once Temporal is more widespread.

The issue

In January 2025 I was in Santa Clara, California writing some JavaScript to perform some reporting. I wanted to be able to get a number of events that happened within a month, so I would create a date object for the first day of the month, add one month to it and then subtract a day to return the last day. Seems straightforward, right?

I got a really weird result though. I reduced the issue to the following code.

const date = new Date("2024-01-01T00:00:00.000Z");
date.toISOString();
// => "2024-01-01T00:00:00.000Z" as expected
date.setMonth(1);
date.toISOString();
// => "2023-03-04-T00:00:00.000Z" WTF?

I added a month to the 1st of January 2024 and landed on the 4th March, 2023. What happened?

Times and zones

You might have thought it was odd for me to set this scene on the West coast of the US, but it turned out this mattered. This code would have run fine in UTC and everywhere East of it.

JavaScript dates are more than just dates, they are responsible for time as well. Even though I only wanted to deal with days and months in this example the time still mattered.

I did know this, so I set the time to UTC thinking that this would work for me wherever I was. That was my downfall. Let's break down what happened.

Midnight on the 1st January, 2024 in UTC is still 4pm on the 31st December, 2023 in Pacific Time (UTC-8). date.setMonth(1) sets the date to February (as months are 0-indexed unlike days). But we started on 31st December, 2023 so JavaScript has to handle the non-existant date of 31st February, 2023. It does this by overflowing to the next month, so we get 3rd March. Finally, to print it out, the date is translated back into UTC, giving the final result: midnight on 4th March, 2023.

All of these steps feel reasonable when you break it down, the confusion stems from how unexpected that result was.

So, how do you fix this?

Always use UTC

Since I didn't actually care for the time and I knew I wanted to work with UTC, I fixed this code using the Date object's setUTCMonth method. My original code subtracted a day to get the last day in a month, so I used the setUTCDate method too. All set${timePeriod} methods have a setUTC${timePeriod} equivalent to help you work with this.

const date = new Date("2024-01-01T00:00:00.000Z");
date.toISOString();
// => "2024-01-01T00:00:00.000Z"
date.setUTCMonth(1);
date.toISOString();
// => "2024-02-01-T00:00:00.000Z"

So this fixed my issue. Can it be better though?

Bring on Temporal

One of the reasons this went wrong was because I was trying to manipulate dates, but I was actually manipulating dates and times without thinking about it. I mentioned Temporal at the top of the post because it has objects specifically for this.

If I was to write this code using Temporal I would be able to use the Temporal.PlainDate to represent a calendar date, a date without a time or time zone.

This simplifies things already, but Temporal also makes it more obvious how to manipulate dates. Rather than setting months and dates or adding milliseconds to update a date, you add a duration. You can either construct a duration with the Temporal.Duration object or use an object that defines a duration.

Temporal also makes objects immutable, so every time you change a date it returns a new object.

In this case I wanted to add a month, so with Temporal it would look like this:

const startDate = Temporal.PlainDate.from("2024-01-01");
// => Temporal.PlainDate 2024-01-01
const nextMonth = startDate.add({ months: 1 });
// => Temporal.PlainDate 2024-02-01
const endDate = nextMonth.subtract({ days: 1 });
// => Temporal.PlainDate 2024-01-31

Date manipulation without worrying about times, wonderful!

Of course, there are many more benefits to the very well throught out Temporal API and I cannot wait for it to be a part of every JavaScript runtime.

Mind the time zone

Temporal has still not made it to many JavaScript engines. At the time of writing, it is available in Firefox and nowhere else, so if you want to test this out open up Firefox or check out one of the polyfills @js-temporal/polyfill or temporal-polyfill.

<div class="info"><p><strong>Edit:</strong> Merely two days after publishing this, <a href="https://developer.chrome.com/blog/new-in-chrome-144#temporal">support for Temporal started rolling out in Chrome 144</a>. According to <a href="https://caniuse.com/temporal">Can I Use, Temporal</a> is available behind a flag in Safari Technical Preview, so we might be close there too.</p></div>

If you still have to use Date make sure you keep your time zone in mind. I'd try to move to, or at least learn how to use, Temporal now.

And watch out for time zones, even when you try to avoid them they can end up giving you a headache.

How to Create Vector Embeddings in Python

Tue, 08 Apr 2025 00:00:00 GMT

When you’re building a retrieval-augmented generation (RAG) app, the first thing you need to do is prepare your data. You need to:

collect your unstructured data
split it into chunks
turn those chunks into vector embeddings
store the embeddings in a vector database

There are many ways that you can create vector embeddings in Python. In this post, we’ll take a look at four ways to generate vector embeddings: locally, via API, via a framework, and with Astra DB's Vectorize.

<div class="info"> <p>This post was originally written for DataStax, but didn't survive a content migration as part of <a href="https://www.ibm.com/new/announcements/ibm-to-acquire-datastax-helping-clients-bring-the-power-of-unstructured-data-to-enterprise-ai-applications">IBM's purchase</a>. I thought the content was useful, so have republished it here.</p> </div>

Local vector embeddings

There are many pre-trained embedding models available on Hugging Face that you can use to create vector embeddings. Sentence Transformers (SBERT) is a library that makes it easy to use these models for vector embedding, as well as cross-encoding for reranking. It even has tools for finetuning models, if that’s something that might be of use.

You can install the library with:

pip install sentence_transformers

A popular local model for vector embedding is all-MiniLM-L6-v2. It’s trained as a good all-rounder that produces a 384-dimension vector from a chunk of text.

To use it, import sentence_transformers and create a model using the identifier from Hugging Face, in this case "all-MiniLM-L6-v2". If you want to use a model that isn't in the sentence-transformers project, like the multilingual BGE-M3, you can use the organization to identify the model too, like, "BAAI/BGE-M3". Once you've loaded the model, use the encode method to create the vector embedding. The full code looks like this:

from sentence_transformers import SentenceTransformer


model = SentenceTransformer("all-MiniLM-L6-v2")
sentence = "A robot may not injure a human being or, through inaction, allow a human being to come to harm."
embedding = model.encode(sentence)

print(embedding)
# => [ 1.95171311e-03  1.51085425e-02  3.36140348e-03  2.48030387e-02 ... ]

If you pass an array of texts to the model, they’ll all be encoded:

from sentence_transformers import SentenceTransformer


model = SentenceTransformer("all-MiniLM-L6-v2")
sentences = [
    "A robot may not injure a human being or, through inaction, allow a human being to come to harm.",
    "A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.",
    "A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.",
]
embeddings = model.encode(sentences)

print(embeddings)
# => [[ 0.00195174  0.01510859  0.00336139 ...  0.07971715  0.09885529  -0.01855042]
# [-0.04523939 -0.00046248  0.02036596 ...  0.08779042  0.04936493  -0.06218244]
# [-0.05453169  0.01125113 -0.00680178 ...  0.06443197  0.08771271  -0.00063468]]

There are many more models you can use to generate vector embeddings with the sentence-transformers library and, because you’re running locally, you can try them out to see which is most appropriate for your data. You do need to watch out for any restrictions that these models might have. For example, the all-MiniLM-L6-v2 model doesn’t produce good results for more than 128 tokens and can only handle a maximum of 256 tokens. BGE-M3, on the other hand, can encode up to 8,192 tokens. However, the BGE-M3 model is a couple of gigabytes in size and all-MiniLM-L6-v2 is under 100MB, so there are space and memory constraints to consider, too.

Local embedding models like this are useful when you’re experimenting on your laptop, or if you have hardware that PyTorch can use to speed up the encoding process. It’s a good way to get comfortable running different models and seeing how they interact with your data.

If you don't want to run your models locally, there are plenty of available APIs you can use to create embeddings for your documents.

APIs

There are several services that make embedding models available as APIs. These include LLM providers like OpenAI, Google, or Cohere, as well as specialist providers like Jina AI or model hosts like Fireworks.

These API providers provide HTTP APIs, often with a Python package to make it easy to call them. You will typically require an API key from the service. Once you have that setup you can generate vector embeddings by sending your text to the API.

For example, with Google's google-genai SDK and a Gemini API key you can generate a vector embedding with their Gemini embedding model like this:

from google import genai


client = genai.Client(api_key="GEMINI_API_KEY")

result = client.models.embed_content(
        model="gemini-embedding-001",
        contents="A robot may not injure a human being or, through inaction, allow a human being to come to harm.")

print(result.embeddings)

Each API can be different, though many providers do make OpenAI-compatible APIs. However, each time you try a new provider you might find you have a new API to learn. Unless, of course, you try one of the available frameworks that are intended to simplify this.

Frameworks

There are several projects available, like LangChain or LlamaIndex, that create abstractions over the common components of the GenAI ecosystem, including embeddings.

Both LangChain and LlamaIndex have methods for creating vector embeddings via APIs or local models, all with the same interface. For example, you can create the same Gemini embedding as the code snippet above with LangChain like this:

from langchain_google_genai import GoogleGenerativeAIEmbeddings


embeddings = GoogleGenerativeAIEmbeddings(
    model="gemini-embedding-001",
    google_api_key="GEMINI_API_KEY"
)
result = embeddings.embed_query("A robot may not injure a human being or, through inaction, allow a human being to come to harm.")
print(result)

As a comparison, here is how you would generate an embedding using an OpenAI embeddings model and LangChain:

from langchain_openai import OpenAIEmbeddings


embeddings = OpenAIEmbeddings(
    model="text-embedding-3-small",
    api_key="OPENAI_API_KEY"
)
result = embeddings.embed_query("A robot may not injure a human being or, through inaction, allow a human being to come to harm.")
print(result)

We had to change the name of the import and the API key we used, but otherwise the code is identical. This makes it easy to swap them out and experiment.

If you're using LangChain to build your entire RAG pipeline, these embeddings fit in well with the vector database interfaces. You can provide an embedding model to the database object and LangChain handles generating the embeddings as you insert documents or perform queries. For example, here's how you can combine the Google embeddings model with the LangChain wrapper for Astra DB.

from langchain_google_genai import GoogleGenerativeAIEmbeddings
from langchain_astradb import AstraDBVectorStore


embeddings = GoogleGenerativeAIEmbeddings(
    model="gemini-embedding-001",
    google_api_key="GEMINI_API_KEY"
)

vector_store = AstraDBVectorStore(
   collection_name="astra_vector_langchain",
   embedding=embeddings,
   api_endpoint="ASTRA_DB_API_ENDPOINT",
   token="ASTRA_DB_APPLICATION_TOKEN"
)

vector_store.add_documents(documents) # a list of document objects to store in the db

You can use the same vector_store object and associated embeddings to perform the vector search, too.

results = vector_store.similarity_search("Are robots allowed to protect themselves?")

LlamaIndex has a similar set of abstractions that enable you to combine different embedding models and vector stores. Check out this LlamaIndex introduction to RAG to learn more.

If you're new to embeddings, LangChain has a handy list of embedding models and providers that can help you find different options to try.

Directly in the database

The methods we’ve talked through so far have involved creating a vector independently of storing it in or using it to search against a vector database. When you want to store those vectors in a vector database like Astra DB, it looks a bit like this:

from astrapy import DataAPIClient


client = DataAPIClient("ASTRA_DB_APPLICATION_TOKEN")
database = client.get_database("ASTRA_DB_API_ENDPOINT")
collection = database.get_collection("COLLECTION_NAME")

result = collection.insert_one(
    {
         "text": "A robot may not injure a human being or, through inaction, allow a human being to come to harm.",
         "$vector": [0.04574034, 0.038084425, -0.00916391, ...]
    }
)

The above assumes that you have already created your vector-enabled collection with the right number of dimensions for the model you’re using.

Performing a vector search then looks like this:

cursor = collection.find(
    {},
    sort={"$vector": [0.04574034, 0.038084425, -0.00916391, ...]}
)

for document in cursor:
    print(document)

In these examples, you have to create your vectors first, before storing or searching against the database with them. In the case of the frameworks, you might not see this happen, as it has been abstracted away, but the operations are being performed.

With Astra DB, you can have the database generate the vector embeddings for you as you either insert the document into the collection or at the point of performing the search. This is called Astra Vectorize and it simplifies a crucial step in your RAG pipeline.

To use Vectorize, you first need to set up an embedding provider integration. There’s one built-in integration that you can use with no extra work; the NVIDIA NV-Embed-QA model, or you can choose one of the other embeddings providers and configure them with your API.

When you create a collection, you can choose which embedding provider you want to use with the requisite number of dimensions.

When you set up your collection this way you can add content and have it automatically vectorized by using the special property $vectorize.

result = collection.insert_one(
    {
         "$vectorize": "A robot may not injure a human being or, through inaction, allow a human being to come to harm."
    }
)

Then, when a user query comes in, you can perform a vector search by sorting using the $vectorize property. Astra DB will create the vector embedding and then make the search in one step.

cursor = collection.find(
    {},
    sort={"$vectorize": "Are robots allowed to protect themselves?"},
    limit=5
)

There are several advantages to this approach:

The Astra DB team has done the work to make the embedding creation robust already
Making two separate API calls to create embeddings and then store them is often slower than letting Astra DB handle it
Using the built-in NVIDIA embeddings model is even quicker than that
You have less code to write and maintain

A world of vector embedding options

As we have seen, there are many choices you can make in how to implement vector embeddings, which model you use, and which provider you use. It's an important step in your RAG pipeline and it is important to spend the time to find out which model and method is right for your application and your data.

You can choose to host your own models, rely on third-party APIs, abstract the problem away through frameworks, or entrust Astra DB to create embeddings for you. Of course, if you want to avoid code entirely, then you can drag-and-drop your components into place with Langflow.

Troubles with multipart form data and fetch in Node.js

Tue, 14 Jan 2025 00:00:00 GMT

This is one of those cathartic blog posts. One in which I spent several frustrating hours trying to debug something that really should have just worked. Once I had finally found out what was going on I felt that I had to write it all down just in case someone else is out there dealing with the same issue. So if you have found yourself in a situation where using fetch in Node.js for a multipart/form-data request doesn't work, this might help you out.

If that doesn't apply to you, have a read anyway and share my pain.

What happened?

Today, while trying to write a Node.js client for an API, I got stuck on one particular endpoint. It was an endpoint for uploading files, so it required the body to be formatted as multipart/form-data. JavaScript makes it easy to create such a request, you use a FormData object to gather your data, including files, and you submit it via fetch. The formatting of the request body is then handled for you and it normally just works.

Today it did not "just work".

HTTP 422

Since version 18, Node.js has supported the fetch API, via a project called undici. The undici project is added as a dependency to Node.js and the fetch function is exposed to the global scope.

To write the code for this upload endpoint should have been straightforward. I put together something like this:

import { readFile } from "node:fs/promises";
import { extname, basename } from "node:path";

async function uploadFile(url, filePath) {
  const data = await readFile(filePath);
  const type = mime.getType(extname(filePath));
  const file = new File([data], basename(filePath), type);

  const form = new FormData();
  form.append("file", file);

  const headers = new Headers();
  headers.set("Accept", "application/json");

  return fetch(url, {
    method: "POST",
    headers,
    body: form
  });
}

The real code has a few more complexities, but this is a good approximation of what I expected to be able to write.

I lined up a test against the API, fired it off and was disappointed to receive a 422 response with the message "Invalid multipart formatting".

I pored back over the code, not that there was a lot of it, to try to work out what I had done wrong. Unable to find anything, I turned to other tools.

I tried to proxy and inspect my request to see if anything was obviously wrong. Then I tried sending the request from another tool to see if I could get the API endpoint to respond with a success. Using Bruno I was able to make a successful request.

With a correct request and an incorrect request, I compared the two. But I didn't get very far. The URL, the headers, and the request body all looked the same, yet one method of sending the request worked and the other didn't.

Digging into the API

The API client I am writing is for Langflow. It's an open-source, low-code tool for building generative AI flows and agents. Langflow is part of DataStax, where I am working and doing things like hooking Langflow up to Bluesky to create fun generative AI bots.

Because Langflow is open-source, once I had run out of ideas with my code I could dig into the code behind the API to see if I could work out what was going on there. I found where the error message was coming from along with a hint as to what might be wrong.

Multipart requests

A multipart request is often made up of multiple parts that may not be of the same type. This is how you are able to submit text fields and upload an image file in the same request. To separate the different types, a multipart request comes up with a unique string to act as a boundary between the types of content. This boundary string is shared in the Content-Type header and looks like this:

Content-Type: multipart/form-data; boundary=ExampleBoundaryString

An example multipart request would then look like this:

POST /foo HTTP/1.1
Content-Length: 68137
Content-Type: multipart/form-data; boundary=ExampleBoundaryString

--ExampleBoundaryString
Content-Disposition: form-data; name="description"

Description input value
--ExampleBoundaryString
Content-Disposition: form-data; name="myFile"; filename="foo.txt"
Content-Type: text/plain

[content of the file foo.txt chosen by the user]
--ExampleBoundaryString--

A server is then able to split up and parse the different parts of the request using the boundary.

This Content-Type example is courtesy of MDN.

The Langflow API was checking to see whether the request body started with the boundary string and ended with the boundary string plus two dashes and then \r\n.

boundary_start = f"--{boundary}".encode()
boundary_end = f"--{boundary}--\r\n".encode()

if not body.startswith(boundary_start) or not body.endswith(boundary_end):
    return JSONResponse(
        status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
        content={"detail": "Invalid multipart formatting"},
    )

The combination of a carriage return and line feed (CRLF), a holdover from the days of typewriters, turned out to be my undoing. After throwing some print lines in this code (I do not know how to debug Python any other way) I confirmed that the end of the body did not match because it was missing the CRLF.

A missing CRLF. Hours of debugging trying to spot a missing \r\n. A Python application causing me trouble over insignificant whitespace.

Convention over specification

It turns out that spec for multipart/form-data, RFC 7578 does not mandate that the body of the request ends with a CRLF. It does say that each different part of the request must be delimited with a CRLF, "--", and then the value of the boundary parameter. It does not say that there needs to be a CRLF at the end of the body, nor does it say that there can't be a CRLF either.

In fact, it turns out that many popular HTTP clients, including curl, do add this CRLF. It's a bit of a convention in HTTP clients, so it seems that when the team at Langflow wanted to implement a check on the validity of a multipart request, they included the CRLF in their expectations.

On the other hand, I can only presume the team building undici looked at the spec and realised they didn't need to add unnecessary whitespace and left the CRLF out.

And this is where I landed. Stuck between an HTTP client that wouldn't add a CRLF and an API that expected it. It took me far too long to figure this out.

Fixing both sides

The latest version of Node.js, as I write this, is 23.6.0 and it still behaves this way. However, the code has been updated in undici version 7.1.0 to include the trailing CRLF and I am sure it will be in a release version of Node.js soon. I'm loathe to call this a fix as there was technically nothing wrong with what they were previously doing, but convention wins here.

On the other side of things, I made a pull request to loosen Langflow's definition of a valid multipart request. I'll have to wait to see how that goes.

As for my own code, I installed the latest version of undici into the project, imported fetch and it started working immediately.

So, if you're using fetch in Node.js between version 18.0.0 and 23.6.0 and you're making requests to a server that expects a multipart request to end in CRLF, you too have felt this pain and I am sorry. Yes, it's specific, but what else is the web for if not for sharing very specific problems and how you eventually mananaged to fix them?

Clean up HTML Content for Retrieval-Augmented Generation with Readability.js

Thu, 09 Jan 2025 00:00:00 GMT

Scraping web pages is one way to fetch content for your retrieval-augmented generation (RAG) application. But parsing the content from a web page can be a pain.

Mozilla's open-source library Readability.js is a useful tool for extracting just the important parts of a web page. Let's look at how to use it as part of a data ingestion pipeline for a RAG application.

Retrieving unstructured data from a web page

Web pages are a source of unstructured data that we can use in RAG-based apps. But web pages are often full of content that is irrelevant; things like headers, sidebars, and footers. They contain useful context for someone browsing the site, but detract from the main subject of a page.

To get the best data for RAG, we need to remove irrelevant content. When you’re working within one site, you can use tools like cheerio to parse the HTML yourself based on your knowledge of the site's structure. But if you're scraping pages across different layouts and designs, you need a good way to return just the relevant content and avoid the rest.

Repurposing reader view

Most web browsers come with a reader view that strips out everything but the article title and content. Here is the difference between the browser and reader mode when applied to a blog post on my personal site.

Mozilla makes the underlying library for Firefox's reader mode available as a standalone open-source module: Readability.js. So we can use Readability.js in a data pipeline to strip irrelevant content and return high quality results from scraping a web page.

How to scrape data with Node.js and Readability.js

Let's take a look at an example of scraping the article content from my previous blog post on creating vector embeddings in Node.js. Here's some JavaScript you can use to retrieve the HTML for the page:

const html = await fetch(
  "https://philna.sh/blog/2024/09/25/how-to-create-vector-embeddings-in-node-js/"
).then((res) => res.text());
console.log(html);

This includes all the HTML tags as well as the navigation, footer, share links, calls to action and other things you can find on most web sites.

To improve on this, you could install a module like cheerio and select only the important parts:

npm install cheerio

import * as cheerio from "cheerio";

const html = await fetch(
  "https://philna.sh/blog/2024/09/25/how-to-create-vector-embeddings-in-node-js/"
).then((res) => res.text());

const $ = cheerio.load(html);

console.log($("h1").text(), "\n");
console.log($("section#blog-content > div:first-child").text());

With this code you get the title and text of the article. As I said earlier, this is great if you know the structure of the HTML, but that won't always be the case.

Instead, install Readability.js and jsdom:

npm install @mozilla/readability jsdom

Readability.js normally runs in a browser environment and uses the live document rather than a string of HTML, so we need to include jsdom to provide that in Node.js. Now we can turn the HTML we already loaded into a document and pass it to Readability.js to parse out the content.

import { Readability } from "@mozilla/readability";
import { JSDOM } from "jsdom";

const url =
  "https://philna.sh/blog/2024/09/25/how-to-create-vector-embeddings-in-node-js/";
const html = await fetch(url).then((res) => res.text());

const doc = new JSDOM(html, { url });
const reader = new Readability(doc.window.document);
const article = reader.parse();

console.log(article);

When you inspect the article, you can see that it has parsed a number of things from the HTML.

There's the title, author, excerpt, publish time, and both the content and textContent. The textContent property is the plain text content of the article, ready for you to split into chunks, create vector embeddings, and ingest into a vector database. The content property is the original HTML, including links and images. This could be useful if you want to extract links or process the images somehow.

You might also want to see whether the document is likely to return good results. Reader view works well on articles, but is less useful for other types of content. You can do a quick check to see if the HTML is suitable for processing with Readability.js with the function isProbablyReaderable. If this function returns false you may want to parse the HTML in a different way, or even inspect the contents at that URL to see whether it has useful content for you.

const doc = new JSDOM(html, { url });
const reader = new Readability(doc.window.document);

if (isProbablyReaderable(doc.window.document)) {
  const article = reader.parse();
  console.log(article);
} else {
  // do something else
}

If the page fails this check, you might want to flag the URL to see whether it does include useful information for your RAG application, or whether it should be excluded.

Using Readability with LangChain.js

If you're using LangChain.js for your application, you can also use Readability.js to return the content from an HTML page. It fits nicely into your data ingestion pipelines, working with other LangChain components, like text chunkers and vector stores.

The following example uses LangChain.js to load the same page as above, return the relevant content from the page using the MozillaReadabilityTransformer, split the text into chunks using the RecursiveCharacterTextSplitter, create vector embeddings with OpenAI, and store the data in Astra DB.

You'll need to install the following dependencies:

npm install @langchain/core @langchain/community @langchain/openai @datastax/astra-db-ts @mozilla/readability jsdom

To run the example, you will need to create an Astra DB database and store the database's endpoint and application token in your environment as ASTRA_DB_APPLICATION_TOKEN and ASTRA_DB_API_ENDPOINT. You will also need an OpenAI API key stored in your environment as OPENAI_API_KEY.

Import the dependencies:

import { HTMLWebBaseLoader } from "@langchain/community/document_loaders/web/html";
import { MozillaReadabilityTransformer } from "@langchain/community/document_transformers/mozilla_readability";
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
import { OpenAIEmbeddings } from "@langchain/openai";
import { AstraDBVectorStore } from "@langchain/community/vectorstores/astradb";

We use the HTMLWebBaseLoader to load the raw HTML from the URL we provide. The HTML is then passed through the MozillaReadabilityTransformer to extract the text, which is then split into chunks by the RecursiveCharacterTextSplitter. Finally, we create an embedding provider and an Astra DB vector store that will be used to turn the text chunks into vector embeddings and store them in the vector database.

const loader = new HTMLWebBaseLoader(
  "https://philna.sh/blog/2024/09/25/how-to-create-vector-embeddings-in-node-js/"
);
const transformer = new MozillaReadabilityTransformer();
const splitter = new RecursiveCharacterTextSplitter({
  maxCharacterCount: 1000,
  chunkOverlap: 200,
});
const embeddings = new OpenAIEmbeddings({
  model: "text-embedding-3-small",
});
const vectorStore = new AstraDBVectorStore(embeddings, {
  token: process.env.ASTRA_DB_APPLICATION_TOKEN,
  endpoint: process.env.ASTRA_DB_API_ENDPOINT,
  collection: "content",
  collectionOptions: {
    vector: {
      dimension: 1536,
      metric: "cosine",
    },
  },
});
await vectorStore.initialize();

The initialisation of all the components makes up most of the work. Once everything is set up, you can load, transform, split, embed and store the documents like this:

const docs = await loader.load();
const sequence = transformer.pipe(splitter);
const vectorizedDocs = await sequence.invoke(docs);
await vectorStore.addDocuments(vectorizedDocs);

More accurate data from web scraping with Readability.js

Readability.js is a battle-tested library powering Firefox's reader mode that we can use to scrape only relevant data from web pages. This cleans up web content and makes it much more useful for RAG.

As we've seen, you can do this directly with the library or using LangChain.js and the MozillaReadabilityTransformer.

Getting data from a web page is only the first step in your ingestion pipeline. From here you'll need to split your text into chunks, create vector embeddings, and store everything in a vector database. Then you'll be ready to build your RAG-powered application.

Shallow clones versus structured clones

Mon, 30 Dec 2024 00:00:00 GMT

Have you ever had one of those times when you think you're doing everything right, yet still you get an unexpected bug in your application? Particularly when it is state-related and you thought you did everything you could to isolate the state by making copies instead of mutating it in place.

Especially when you are, say, building a game that copies a blank initial state when you create a new room and, no matter what you do, you still find that every player is in every room.

If you find yourself in this sort of situation, like I might have recently, then it's almost certain that you have nested state, you are only making a shallow clone, and you should be using structuredClone.

Shallow copies of nested states

Here's a simple version of the issue I described above. We have a default state and when we generate a new room that state is cloned to the room. The room has a function to add a player to its state.

const defaultState = {
  roomName: "",
  players: []
}

class Room {
  constructor(name) {
    this.state = { ...defaultState, roomName: name }
  }

  addPlayer(playerName) {
    this.state.players.push(playerName);
  }
}

You can create a new room and add a player to it.

const room = new Room("room1");
room.addPlayer("Phil");
console.log(room.state);
// { roomName: "room1", players: ["Phil"] }

But if you try to create a second room, you'll find the player is already in there.

const room2 = new Room("room2");
console.log(room2.state);
// { roomName: "room2", players: ["Phil"] }

It turns out the player even entered the default state.

console.log(defaultState);
// { roomName: "", players: ["Phil"] }

The issue is the clone of the default state that was made in the constructor. It uses object spread syntax (though it could have been using Object.assign) to make a shallow clone of the default state object, but a shallow clone is only useful for cloning primitive values in the object. If you have values like arrays or objects, they aren't cloned, the reference to the original object is cloned.

You can see this because the players array in the above example is equal across the default state and the two rooms.

defaultState.players === room.state.players;
// true
defaultState.players === room2.state.players;
// true
room.state.players === room2.state.players;
// true

Since all these references point to the same object, whenever you make an update to any room's players, all rooms and the default state will reflect that.

How to make a deep clone

There have been many ways in JavaScript over the years to make deep clones of objects, examples include Lodash's cloneDeep and using JSON to stringify and then parse an object. However it turned out that the web platform already had an underlying algorithm to perform deep clones through APIs like postMessage.

In 2015 it was suggested on a W3C mailing list that the algorithm was exposed publicly, though it took until late 2021 when Deno, Node.js and Firefox released support for structuredClone, followed in 2022 by the other browsers.

If you want to make a deep clone of an object in JavaScript you should use structuredClone.

Using structuredClone

Let's see the function in action. If we update the Room class from the example above to use structuredClone, it looks like this:

class Room {
  constructor(name) {
    this.state = structuredClone(defaultState);
    this.state.roomName = name;
  }

  addPlayer(playerName) {
    this.state.players.push(playerName);
  }
}

Creating one room acts as it did before:

const room = new Room("room1");
room.addPlayer("Phil");
console.log(room.state);
// { roomName: "room1", players: ["Phil"] }

But creating a second room now works as expected, the players are no longer shared.

const room2 = new Room("room2");
console.log(room2.state);
// { roomName: 'room2', players: [] }

And the players arrays are no longer equal references, but completely different objects:

defaultState.players === room.state.players;
// false
defaultState.players === room2.state.players;
// false
room.state.players === room2.state.players;
// false

It is worth reading how the structured clone algorithm works and what doesn't work. For example, you cannot deep clone Function objects or DOM nodes, but circular references are handled with no problem.

What a state

If you find yourself in a pickle when trying to clone your state, it may be because you are only making a shallow clone. If you have a simple state that is made up of primitives, then it is fine to use a shallow clone. Once you introduce nested objects that's when you need to consider using structuredClone to create a deep clone and avoid just copying references.

If you do find yourself facing this issue, I hope it takes you less time than it took me to realise what was going on.

How to Create Vector Embeddings in Node.js

Wed, 25 Sep 2024 00:00:00 GMT

When you’re building a retrieval-augmented generation (RAG) app, job number one is preparing your data. You’ll need to take your unstructured data and split it up into chunks, turn those chunks into vector embeddings, and finally, store the embeddings in a vector database.

There are many ways that you can create vector embeddings in JavaScript. In this post, we’ll investigate four ways to generate vector embeddings in Node.js: locally, via API, via a framework, and with Astra DB's Vectorize.

Local vector embeddings

There are lots of open-source models available on HuggingFace that can be used to create vector embeddings. Transformers.js is a module that lets you use machine learning models in JavaScript, both in the browser and Node.js. It uses the ONNX runtime to achieve this; it works with models that have published ONNX weights, of which there are plenty. Some of those models we can use to create vector embeddings.

You can install the module with:

npm install @huggingface/transformers

The package can actually perform many tasks, but feature extraction is what you want for generating vector embeddings.

A popular, local model for vector embedding is all-MiniLM-L6-v2. It’s trained as a good all-rounder and produces a 384-dimension vector from a chunk of text.

To use it, import the pipeline function from Transformers.js and create an extractor that will perform "feature-extraction" using your provided model. You can then pass a chunk of text to the extractor and it will return a tensor object which you can turn into a plain JavaScript array of numbers.

All in all, it looks like this:

import { pipeline } from "@huggingface/transformers";

const extractor = await pipeline(
  "feature-extraction",
  "Xenova/all-MiniLM-L6-v2"
);

const response = await extractor(
  [
    "A robot may not injure a human being or, through inaction, allow a human being to come to harm.",
  ],
  { pooling: "mean", normalize: true }
);

console.log(Array.from(response.data));
// => [-0.004044221248477697,  0.026746056973934174,   0.0071970801800489426, ... ]

You can actually embed multiple texts at a time if you pass an array to the extractor. Then you can call tolist on the response and that will return you a list of arrays as your vectors.

const response = await extractor(
  [
    "A robot may not injure a human being or, through inaction, allow a human being to come to harm.",
    "A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.",
    "A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.",
  ],
  { pooling: "mean", normalize: true }
);

console.log(response.tolist());
// [
//   [ -0.006129210349172354,  0.016346964985132217,   0.009711502119898796, ...],
//   [-0.053930871188640594,  -0.002175076398998499,   0.032391052693128586, ...],
//   [-0.05358131229877472,  0.021030642092227936, 0.0010665050940588117, ...]
// ]

There are many models you can use to create vector embeddings from text, and, because you’re running locally, you can try them out to see which works best for your data. You should pay attention to the length of text that these models can handle. For example, the all-MiniLM-L6-v2 model does not provide good results for more than 128 tokens and can handle a maximum of 256 tokens, so it’s useful for sentences or small paragraphs. If you have a bigger source of text data than that, you’ll need to split your data into appropriately sized chunks.

Local embedding models like this are useful if you’re experimenting on your own machine, or have the right hardware to run them efficiently when deployed. It's an easy way to get comfortable with different models and get a feel for how things work without having to sign up to a bunch of different API services.

Having said that, there are a lot of useful vector embedding models available as an API, so let's take a look at them next.

APIs

There are an abundance of services that provide embedding models as APIs. These include LLM providers, like OpenAI, Google or Cohere, as well as specialist providers like Voyage AI or Jina. Most providers have general purpose embedding models, but some provide models trained for specific datasets, like Voyage AI's finance, law and code optimised models.

These API providers provide HTTP APIs, often with an npm package to make it easy to call them. You’ll typically need an API key from the service and you can then generate embeddings by sending your text to the API.

For example, you can use Google's text embedding models through the Gemini API like this:

import { GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: process.env.API_KEY });
const text =
  "A robot may not injure a human being or, through inaction, allow a human being to come to harm.";

const response = await ai.models.embedContent({
  model: "gemini-embedding-001",
  contents: text,
});

console.log(response.embeddings[0].values);
// => [ -0.0049246787, 0.031826325, -0.0075687882, ... ]

Each API is different though, so while making a request to create embeddings is normally fairly straightforward, you’ll likely have to learn a new method for each API you want to call—unless of course, you try one of the available frameworks that are intended to simplify this.

Frameworks

There are many projects out there, like LangChain or LlamaIndex, that create abstractions over the various parts of the GenAI toolchain, including embeddings.

Both LangChain and LlamaIndex enable you to generate embeddings via APIs or local models, all with the same interface. For example, here’s how you can create the same embedding as above using the Gemini API and LangChain together:

import { GoogleGenerativeAIEmbeddings } from "@langchain/google-genai";

const embeddings = new GoogleGenerativeAIEmbeddings({
  apiKey: process.env.API_KEY,
  model: "gemini-embedding-001",
});
const text =
  "A robot may not injure a human being or, through inaction, allow a human being to come to harm.";

const embedding = await embeddings.embedQuery(text);
console.log(embedding);
// => [-0.0049246787, 0.031826325, -0.0075687882, ...]

To compare, this is what it looks like to use the OpenAI embeddings model through LangChain:

import { OpenAIEmbeddings } from "@langchain/openai";

const embeddings = new OpenAIEmbeddings({
  apiKey: process.env.API_KEY,
  model: "text-embedding-3-large",
});
const text =
  "A robot may not injure a human being or, through inaction, allow a human being to come to harm.";

const embedding = await embeddings.embedQuery(text);
console.log(embedding);
// => [0.009445431, -0.0073068426, -0.00814802, ...]

Aside from changing the name of the import and sometimes the options, the embedding models all have a consistent interface to make it easier to swap them out.

If you’re using LangChain to create your entire pipeline, these embedding interfaces work very well alongside the vector database interfaces. You can provide an embedding model to the database integration and LangChain handles generating the embeddings as you insert documents or perform vector searches. For example, here is how to embed some documents using Google's embeddings and store them in Astra DB via LangChain:

import { GoogleGenerativeAIEmbeddings } from "@langchain/google-genai";
import { AstraDBVectorStore } from "@langchain/community/vectorstores/astradb";

const embeddings = new GoogleGenerativeAIEmbeddings({
  apiKey: process.env.API_KEY,
  model: "gemini-embedding-001",
});

const vectorStore = await AstraDBVectorStore.fromDocuments(
  documents, // a list of document objects to put in the store
  embeddings, // the embeddings model
  astraConfig // config to connect to Astra DB
);

When you provide the embeddings model to the database object, you can then use it to perform vector searches too.

const results = vectorStore.similaritySearch(
  "Are robots allowed to protect themselves?"
);

LlamaIndex allows for similar creation of embedding models and vector stores that use them. Check out the LlamaIndex documentation on RAG.

As a bonus, the lists of models that LangChain and LlamaIndex integrate are good examples of popular embedding models.

Directly in the database

So far, the methods above mostly involve creating a vector embedding independently of storing the embedding in a vector database. When you want to store those vectors in a vector database like Astra DB, it looks a bit like this:

import { DataAPIClient } from "@datastax/astra-db-ts";
const client = new DataAPIClient(process.env.ASTRA_DB_APPLICATION_TOKEN);
const db = client.db(process.env.ASTRA_DB_API_ENDPOINT);
const collection = db.collection(process.env.ASTRA_DB_COLLECTION);

await collection.insertOne({
  text: "A robot may not injure a human being or, through inaction, allow a human being to come to harm.",
  $vector: [0.04574034, 0.038084425, -0.00916391, ...]
});

This assumes you have already created a vector enabled collection with the correct number of dimensions for the model you are using.

You can also search against the documents in your collection using a vector like this:

const cursor = collection.find({}, {
  sort: { $vector: [0.04574034, 0.038084425, -0.00916391, ...] },
  limit: 5,
});
const results = await cursor.toArray();

In this case, you have to create your vectors first, and then store or search against the database with them. Even in the case of the frameworks, that process happens, but it’s just abstracted away.

With Astra DB, you can have the database generate the embeddings for you as you’re inserting documents into a collection or as you perform a vector search against a collection.

This is called Astra DB vectorize; here's how it works.

First, set up an embedding provider integration. There is a built-in integration offering the NVIDIA NV-Embed-QA model, or you can choose one of the other providers and configure them with your own API key.

Then when you set up a collection, you can choose which embedding provider you want to use and set the correct number of dimensions.

Now, when you add a document to this collection, you can add the content using the special key $vectorize and a vector embedding will be created.

await collection.insertOne({
  $vectorize:
    "A robot may not injure a human being or, through inaction, allow a human being to come to harm.",
});

When you want to perform a vector search against this collection, you can sort by the special $vectorize field and again, Astra DB will handle creating vector embeddings and then performing the search.

const cursor = collection.find(
  {},
  {
    sort: { $vectorize: "Are robots allowed to protect themselve?" },
    limit: 5,
  }
);
const results = await cursor.toArray();

This has several advantages:

It's robust, as Astra DB handles the interaction with the embedding provider
It can be quicker than making two separate API calls to create embeddings and then store them
It's less code for you to write

Choose the method that works best for your application

There are many models, providers, and methods you can use to turn text into vector embeddings. Creating vector embeddings from your content is a vital part of the RAG pipeline and it does require some experimentation to get it right for your data.

You have the choice to host your own models, call on APIs, use a framework, or let Astra DB handle creating vector embeddings for you. And, if you want to avoid code altogether, you could choose to use Langflow's drag-and-drop interface to create your RAG pipeline.

How to Chunk Text in JavaScript for Your RAG Application

Wed, 18 Sep 2024 00:00:00 GMT

Retrieval-augmented generation (RAG) applications begin with data, so getting your data in the right shape to work well with vector databases and large language models (LLMs) is the first challenge you’re likely to face when you get started building. In this post, we'll discuss the different ways to work with text data in JavaScript, exploring how to split it up into chunks and prepare it for use in a RAG app.

Why chunking is important

Often you will have swaths of unstructured text content that need to be broken down into smaller chunks of data. These chunks of text are turned into vector embeddings and stored in a vector database like Astra DB.

Compared to a whole document, smaller chunks of text capture fewer topics or ideas, which means that their embeddings will contain more focused meaning. This makes each chunk easier to match against incoming user queries and makes for more accurate retrieval. When you improve your retrieval, you can feed fewer, but more relevant, tokens to an LLM and create a more accurate and useful RAG system.

If you want to read more on the theory behind text chunking check out this post from Unstructured on best practices for chunking.

Chunking in JavaScript

Let's move beyond theory and take a look at the options in JavaScript to chunk your data. The libraries we're going to look at are: llm-chunk, LangChain, LlamaIndex, and semantic-chunking. You can experiment with these chunking libraries using this app I put together.

We'll also take a look at using the Unstructured API for more complex use cases. Let's get started with the simplest of these modules.

llm-chunk

llm-chunk describes itself as a "super simple and easy-to-use text splitter" and it is! You can install it using npm with:

npm install llm-chunk

It consists of one function, chunk, and it takes just a few options. You can choose maximum and minimum sizes for the chunks it produces, pick the size of the overlap between chunks, and choose a strategy for splitting the text, either by sentence or paragraph.

By default, it will split text up by paragraphs with a maximum length of 1,000 characters, and no minimum length or overlap.

import { chunk } from "llm-chunk";
const text = loadText(); // get some text to split
const chunks = chunk(text);

You can choose to split the text by sentences, alter the maximum and minimum number of characters by chunk, or by how many characters each chunk should overlap.

const chunks = chunk(text, { minLength: 128, maxLength: 1024, overlap: 128 });

For more complex use cases, you can opt to parse a set of delimiters. These get turned into a regular expression and used as the basis for splitting up the text instead of just by paragraph or sentence. llm-chunk is simple, fast, and a great option when you are starting your RAG journey.

LangChain

LangChain is much more than a text splitter. It is a library that helps you load data, split it up, embed it, store that data in and retrieve it from vector databases, feed it as a prompt to LLMs, and more. There’s a lot to explore with LangChain, but we’re going to concentrate on text splitting.

You can install the LangChain text splitters with this command:

npm install @langchain/textsplitters

If you are using the main langchain module then @langchain/textsplitters is included as a dependency.

LangChain includes three main splitter classes, the CharacterTextSplitter, RecursiveCharacterTextSplitter, and TokenTextSplitter. Let's take a look at what each of them do.

`CharacterTextSplitter`

This is the simplest of the splitters provided by LangChain. It splits up a document by a character, then merges segments back together until they reach the desired chunk size, overlapping the chunks by the desired number of characters.

The default character for splitting up text is "\n\n". This means it aims to initially split the text by paragraphs, though you can also provide the character you want to split by too. Here's how you would use this LangChain splitter:

import { CharacterTextSplitter } from "@langchain/textsplitters";
const text = loadText(); // get some text to split
const splitter = new CharacterTextSplitter({
  chunkSize: 1024,
  chunkOverlap: 128,
});
const output = await splitter.splitText(text);

You can also output LangChain Documents which is useful if you are using the rest of LangChain to create a data pipeline:

const output = await splitter.createDocuments([text]);

`RecursiveCharacterTextSplitter`

The CharacterTextSplitter is naive, and doesn't take into account much of the structure of a piece of text. The RecursiveCharacterTextSplitter goes beyond by using a list of separators to progressively break text down until it creates chunks that fit the size you want. By default it splits text first by paragraphs, then sentences, then words.

import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
const text = loadText(); // get some text to split
const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1024,
  chunkOverlap: 128,
});
const output = splitter.splitText(text);

You can provide different characters for the RecursiveCharacterTextSplitter to split the text on so it can be used to split up other types of text. There are two classes available that make it easy to split up Markdown or Latex, the MarkdownTextSplitter and LatexTextSplitter respectively.

But this can also be used to split up code for a number of different languages. For example, if you wanted to split up a JavaScript file, you could do this:

const splitter = RecursiveCharacterTextSplitter.fromLanguage("js", {
  chunkSize: 300,
  chunkOverlap: 0,
});
const jsOutput = await splitter.splitText(jsCode);

The RecursiveCharacterTextSplitter is a versatile text splitter and is likely a good first stop when you're building up a pipeline to ingest data for your RAG application.

TokenTextSplitter

Taking a different approach, the TokenTextSplitter turns the text first into tokens using js-tiktoken, splits the tokens into chunks and then converts the tokens back into text.

Tokens are the way that LLMs consume content, a token can be a whole word or just a part of it. OpenAI has a good representation of how their models break text into tokens. You can use the TokenTextSplitter like this:

import { TokenTextSplitter } from "@langchain/textsplitters";
const text = loadText(); // get some text to split
const splitter = new TokenTextSplitter({
  chunkSize: 1024,
  chunkOverlap: 128,
});
const output = splitter.splitText(text);

LlamaIndex

LlamaIndex is also responsible for much more than just text splitting. We're going to hone in on the splitting capabilities though, which you can use outside of LlamaIndex too.

LlamaIndex considers chunks of a document as Nodes and the rest of the library works with Nodes. There are three available processors: SentenceSplitter, MarkdownNodeParser. and SentenceWindowNodeParser.

You can install the entire LlamaIndex suite with:

npm install llamaindex

If you just want the text parsers, you can install just the LlamaIndex core:

npm install @llamaindex/core

`SentenceSplitter`

The SentenceSplitter is the simplest of the LlamaIndex splitters. It splits the text into sentences and then combines them into a string that is smaller than the provided chunkSize.

LlamaIndex does this differently from the previous splitters; it measures the size of a chunk in tokens rather than characters. There are approximately four characters to a token and a good default is 1024 characters with an overlap of 128, so you should aim for chunks of 256 tokens and an overlap of 32 tokens.

The SentenceSplitter returns an array of chunks like this:

import { SentenceSplitter } from "@llamaindex/core/node-parser";
const text = loadText(); // get some text to split
const splitter = new SentenceSplitter({
  chunkSize: 256,
  chunkOverlap: 32,
});
const output = splitter.splitText(text);

`MarkdownNodeParser`

If you have Markdown to split up then the MarkdownNodeParser will split it up into logical sections based on the headers in the document.

This splitter doesn't let you set a chunkSize or overlap, so you do give up that level of control. It also works over LlamaIndex Documents or Nodes. In this example we turn our text into a Document first, then get the Nodes from the Documents.

import { MarkdownNodeParser } from "@llamaindex/core/node-parser";
import { Document } from "@llamaindex/core/schema";
const text = loadText(); // get some text to split
const splitter = new MarkdownNodeParser();
const nodes = splitter.getNodesFromDocuments([new Document({ text })]);
const output = nodes.map((node) => node.text);

`SentenceWindowNodeParser`

The final LlamaIndex parser breaks text down into sentences and then produces a Node for each sentence with a window of sentences to either side. You can choose how big the window is. Choosing a window size of one will produce Nodes with three sentences, the current sentence, one before and one after. A window size of two produces Nodes with five sentences, the current sentence, and two either side.

This parser works on Documents as well; you use it like so:

import { SentenceWindowNodeParser } from "@llamaindex/core/node-parser";
import { Document } from "@llamaindex/core/schema";
const text = loadText(); // get some text to split
const splitter = new SentenceWindowNodeParser({ windowSize: 3 });
const nodes = splitter.getNodesFromDocuments([new Document({ text })]);
const output = nodes.map((node) => node.text);

semantic-chunking

semantic-chunking is not a popular text splitter in the JavaScript world, but I wanted to include it as something that’s a bit different. It still gives you control over the maximum size of your chunks, but the way it splits up the chunks uses a more interesting method.

It first splits the text into sentences, and then it generates embedding vectors for each sentence using a locally downloaded model (by default it uses Xenova/all-MiniLM-L6-v2 but you can choose a different one if you want). It then groups the sentences into chunks based on how similar they are using cosine similarity. The intention is to group sentences that go together and have related contents, and then, when the topic changes, start a new chunk.

This is a smarter type of chunking than just splitting by chunk size, and likely even smarter than the markdown parsers that at least take section headings into account. The trade-off is that it is likely to be slower as there is more computation to be done.

It is still simple to use though; install it with:

npm install semantic-chunking

You can pass a similarityThreshold to the chunkit function, which is the minimum cosine similarity required for two sentences to be included in the same chunk. A high threshold provides a high bar for a sentence to be included and likely results in smaller chunks. A low threshold will allow for fewer and bigger chunks. As always, it’s worth experimenting with this setting to find what works for your data.

import { chunkit } from "semantic-chunking";
const text = loadText(); // get some text to split
const chunks = chunkit(text, {
  maxTokenSize: 256,
  similarityThreshold: 0.5,
});

There are other options around further combining chunks, check out the documentation for more detail.

Unstructured

Unstructured is a platform for extracting data from files and chunking it in a smart way. There is an open-source toolkit for this, a no-code platform, and an API. We're going to investigate the API here.

The API has support for extracting data from loads of different file types, including images and documents like PDFs that may contain images or tables of data. Below is a simple example of calling the Unstructured API; you can read about more of the capabilities, particularly around extracting data from PDFs and images, in the Unstructured API documentation.

First you should install the API client:

npm install unstructured-client

You will need an API key. You can get a free API key, or the paid service has a two-week free trial. Once you have an API key, you can use the client to call the API.

In this example, I am chunking text from a markdown file, but check out the API parameters you can use for behaviour with other types of file.

import { UnstructuredClient } from "unstructured-client";
import { ChunkingStrategy } from "unstructured-client/sdk/models/shared/index.js";
import { readFileSync } from "node:fs";

const client = new UnstructuredClient({
  serverURL: "https://api.unstructuredapp.io",
  security: {
    apiKeyAuth: "YOUR_API_KEY_HERE",
  },
});

const data = readFileSync("./post.md");

const res = await client.general.partition({
  partitionParameters: {
    files: {
      content: data,
      fileName: "post.md",
    },
    chunkingStrategy: ChunkingStrategy.BySimilarity,
  },
});
if (res.statusCode == 200) {
  console.log(res.elements);
}

You'll notice I selected a chunking strategy of similarity; you can read about the other chunking strategies available in the documentation as well as partitioning strategies, which are used for documents like images and PDFs.

What's your chunking strategy?

It turns out that there are many options for splitting text up into chunks in the JavaScript ecosystem. From the quick and simple, like llm-chunk, fully featured libraries with full GenAI pipelines like LangChain or LlamaIndex, and more complex methods like semantic-chunking and the Unstructured API with all the options it brings for loading and splitting documents.

Getting your chunking strategy right is important for getting good results from your RAG application, so ensure you read about the best practices and try out the available options to see the sort of results you get from them. If you want to experiment with the libraries llm-chunk, LangChain, LlamaIndex, and semantic-chunking, check out the example application, Chunkers.

However you turn your text into chunks for your RAG application, it's good to understand all of the available options.

How Using Fetch with the Streams API Gets You Faster UX with GenAI Apps

Thu, 22 Aug 2024 00:00:00 GMT

Generative AI enables us to build incredible new types of applications, but large language model (LLM) responses can be slow. If we wait for the full response before updating the user interface, we might be making our users wait more than they need to. Thankfully, most LLM APIs—including OpenAI, Anthropic, and Langflow provide streaming endpoints that you can use to stream responses as they are generated. In this post, we're going to see how to use JavaScript's fetch API to immediately update your front-end application as an LLM generates output and create a better user experience.

Slow responses without streaming

Let's start with an example to show what a slow result looks like to a user. You can try it here. This example GitHub repo demonstrates an Express application that serves up static files and has one endpoint that streams some lorem ipsum text to the front-end. The following code is how the server streams the response:

app.get("/stream", async (_req, res) => {
  res.set("Content-Type", "text/plain");
  res.set("Transfer-Encoding", "chunked");

  const textChunks = text.replace(/\n/g, "<br>").split(/(?<=\.)/g);

  for (let chunk of textChunks) {
    res.write(chunk);
    await sleep(250);
  }

  res.end();
});

The Content-Type header shows that we are returning text and the Transfer-Encoding header tells the browser that the text will be arriving as a stream of chunks.

Express enables you to write content to the response at any time, using res.write. In this case, we break a section of text up into sentences and then write each sentence after a 250 millisecond gap. This is standing in for our streaming LLM response, as it is simpler than putting together an entire GenAI application to demonstrate this.

Let's take a look at how we would normally implement a fetch request to get some data.

const response = await fetch("/stream");
const text = await response.text();
output.innerHTML = text;

The text function collects the entire response then decodes it into text. In this example, we write the text to the page. This is fine if the response is fast, but our example server above is returning a new chunk of text every quarter of a second, and it will depend on how long the response is as to how long it takes to resolve response.text().

Streaming with fetch

Setting up to stream a response from a server is not as straightforward, but the results are worth it. We're going to use the fact that the body of a response is a ReadableStream. We can then set up a streaming pipeline that decodes the incoming stream and writes it to the page.

We need to decode the stream because the chunks of stream we receive are bytes in the form of a Uint8Array. We want them as text, so we can use a TextDecoderStream to decode the bytes as they flow through. The TextDecoderStream is an implementation of a TransformStream; it has both a readable and writable stream, so it can be used in streaming pipelines.

We then want to write the text to the page, so we can build a custom WritableStream to handle that. Let's see the code:

const response = await fetch("/stream");
const decoderStream = new TextDecoderStream("utf-8");
const writer = new WritableStream({
  write(chunk) {
    output.innerHTML += chunk;
  },
});

response.body.pipeThrough(decoderStream).pipeTo(writer);

And now let's compare this version side-by-side to the version that waits for the whole response before it decodes and writes it to the page.

As you can see, the text takes the same amount of time to load whether we use the streaming version or the regular version, but users can start reading the response much earlier when it streams straight to the page.

When you are dealing with LLMs, or other responses that progressively generate a response, streaming the response and rendering it to the page as you receive it gives users a much better perceived performance over waiting for the full response.

More to know about streams

There's more to streams than the code above. Our basic implementation of a WritableStream above included a write function that is called when new chunks of data are ready to be written. You can also define the following functions:

start: called as soon as the stream is constructed and used to set up any resources needed for writing the data
close: called once there are no more chunks to write to the stream and used to release any resources
abort: called if the application signals that the stream should be immediately closed and, like close, used to clean up any resources. Unlike close, abort will run even if there are chunks still to be written and cause them to be thrown away.

You can also pass a queuingStrategy to a WritableStream that allows you to control how fast the stream receives data. This is known as "backpressure," and you can read more about it on MDN.

Catching errors

If there is an error in the stream, like a break in the connection between the front-end and the server, you will need to catch the error. The stream method pipeTo is the final method that is called in a pipeline and returns a Promise. You can either catch errors with the catch method on the Promise:

response.body
  .pipeThrough(decoderStream)
  .pipeTo(writer)
  .catch((error) => {
    console.log("Something went wrong with the stream!");
  });

Or you can await the result and use a try/catch block:

try {
  await response.body
    .pipeThrough(decoderStream)
    .pipeTo(writer);
} catch((error) => {
  console.log("Something went wrong with the stream!");
}

Server-sent events

In the example above, the stream from the server was just text. Many LLM APIs, including Anthropic, Google, OpenAI, and Langflow, send more data back than just the text response and use the server-sent events standard to format those streams.

To send data as server-sent events we'd need to change a couple of things from our original server. The Content-Type becomes text/event-stream. Instead of sending plain text, each message must be labeled as either "event," "data," "id," or "retry," and messages are separated by two newlines. Taking the server side example from earlier, we could update it like so:

app.get("/stream", async (_req, res) => {
  res.set("Content-Type", "text/event-stream");
  res.set("Transfer-Encoding", "chunked");

  const textChunks = text.replace(/\n/g, "<br>").split(/(?<=\.)/g);

  for (let chunk of textChunks) {
    res.write(`event: message\ndata: ${chunk}\n\n`);
    await sleep(250);
  }

  res.end();
});

Now each message is of the form:

event: message
data: ${textChunk}

Which means we need to parse this on the client side. Normally, this is easily done by making a connection to the server using the EventSource object and letting the browser parse the messages into events. If you are building with GenAI, you're most likely sending user queries to the server over a POST request, which is not supported by EventSource.

To use server-sent events with a POST request, you’ll need to parse the responses. One way to do this is to use the eventsource-parser module, which even makes a transform stream available; this fits in nicely with our existing application.

To handle server-sent events in our existing front-end, we can import an EventSourceParserStream from eventsource-parser/stream and use it in our pipeline.

const response = await fetch("/stream");
const decoderStream = new TextDecoderStream("utf-8");
const parserStream = new EventSourceParserStream();
const writer = new WritableStream({
  write(event) {
    output.innerHTML += event.data;
  },
});

response.body
  .pipeThrough(decoderStream)
  .pipeThrough(parserStream)
  .pipeTo(writer);

Note that now we get an event emitted from the stream that has a data property containing the data sent from the server. This is still text in this application, but you could send, for example, a JSON object containing structured data instead. You can see this implementation of server-sent event streaming in the example GitHub repo.

Async iterables

There is an easier way to handle the incoming chunks of a streaming response using a for await...of loop.

In this case we don't need to create our own WritableStream, though we do still need to use a TextDecoder to decode the bytes into a string. It relies on the body of the response being a ReadableStream that implements the async iterable protocol, and looks like this:

const decoder = new TextDecoder();
for await (const chunk of response.body) {
  const text = decoder.decode(chunk, { stream: true });
  streamOutput.innerHTML += text;
}

Sadly, Safari does not support this technique, so we have to avoid it in the front-end for now.

Get comfortable with streaming

Understanding how to stream a response from a web server and consume it on the client is vital to creating a great user experience when working with LLMs. Using fetch as a ReadableStream is the web-native way to do so. When you stream a potentially slow text response, you can improve the perceived performance of your application, leading to happier users and customers

If you're playing with Langflow, you'll find that you can stream responses from the API, and you should take advantage of this.

Vercel's AI SDK uses fetch to make streaming easier for a number of frameworks with the useChat hook. For React Server Component users, you can also stream UI using Vercel's AI RSC API.

If you want to play around with streaming, you can check out the example code from this blog post on GitHub.

JavaScript is getting array grouping methods

Thu, 14 Sep 2023 00:00:00 GMT

Grouping items in an array is one of those things you've probably done a load of times. Each time you would have written a grouping function by hand or perhaps reached for lodash's groupBy function.

The good news is that JavaScript is now getting grouping methods so you won't have to anymore. Object.groupBy and Map.groupBy are new methods that will make grouping easier and save us time or a dependency.

Grouping until now

Let's say you have an array of objects representing people and you want to group them by their age. You might use a forEach loop like this:

const people = [
  { name: "Alice", age: 28 },
  { name: "Bob", age: 30 },
  { name: "Eve", age: 28 },
];

const peopleByAge = {};

people.forEach((person) => {
  const age = person.age;
  if (!peopleByAge[age]) {
    peopleByAge[age] = [];
  }
  peopleByAge[age].push(person);
});
console.log(peopleByAge);
/*
{
  "28": [{"name":"Alice","age":28}, {"name":"Eve","age":28}],
  "30": [{"name":"Bob","age":30}]
}
*/

Or you may choose to use reduce, like this:

const peopleByAge = people.reduce((acc, person) => {
  const age = person.age;
  if (!acc[age]) {
    acc[age] = [];
  }
  acc[age].push(person);
  return acc;
}, {});

Either way, it's slightly awkward code. You always have to check the object to see whether the grouping key exists and if not, create it with an empty array. Then you can push the item into the array.

Grouping with Object.groupBy

With the new Object.groupBy method, you can outcome like this:

const peopleByAge = Object.groupBy(people, (person) => person.age);

Much simpler! Though there are some things to be aware of.

Object.groupBy returns a null-prototype object. This means the the object does not inherit any properties from Object.prototype. This is great because it means you won't accidentally overwrite any properties on Object.prototype, but it also means that the object doesn't have any of the methods you might expect, like hasOwnProperty or toString.

const peopleByAge = Object.groupBy(people, (person) => person.age);
console.log(peopleByAge.hasOwnProperty("28"));
// TypeError: peopleByAge.hasOwnProperty is not a function

The callback function you pass to Object.groupBy should return a string or a Symbol. If it returns anything else, it will be coerced to a string.

In our example, we have been returning the age as a number, but in the result it is coerced to string. Though you can still access the properties using a number as using square bracket notation will also coerce the argument to string.

console.log(peopleByAge[28]);
// => [{"name":"Alice","age":28}, {"name":"Eve","age":28}]
console.log(peopleByAge["28"]);
// => [{"name":"Alice","age":28}, {"name":"Eve","age":28}]

Grouping with Map.groupBy

Map.groupBy does almost the same thing as Object.groupBy except it returns a Map. This means that you can use all the usual Map functions. It also means that you can return any type of value from the callback function.

const ceo = { name: "Jamie", age: 40, reportsTo: null };
const manager = { name: "Alice", age: 28, reportsTo: ceo };

const people = [
  ceo,
  manager,
  { name: "Bob", age: 30, reportsTo: manager },
  { name: "Eve", age: 28, reportsTo: ceo },
];

const peopleByManager = Map.groupBy(people, (person) => person.reportsTo);

In this case, we are grouping people by who they report to. Note that to retrieve items from this Map by an object, the objects have to have the same identity.

peopleByManager.get(ceo);
// => [{ name: "Alice", age: 28, reportsTo: ceo }, { name: "Eve", age: 28, reportsTo: ceo }]
peopleByManager.get({ name: "Jamie", age: 40, reportsTo: null });
// => undefined

In the above example, the second line uses an object that looks like the ceo object, but it is not the same object so it doesn't return anything from the Map. To retrieve items successfully from the Map, make sure you keep a reference to the object you want to use as the key.

When will this be available?

The two groupBy methods are part of a TC39 proposal that is currently at stage 3. This means that there is a good chance it will become a standard and, as such, there are implementations appearing.

Chrome 117 just launched with support for these two methods and Firefox released support in version 119. Safari had implemented these methods under different names, I'm sure they will update that soon. As the methods are in Chrome that means they have been implemented in V8, so will be available in Node the next time V8 is updated.

Why use static methods?

You might wonder why this is being implemented as Object.groupBy and not Array.prototype.groupBy. According to the proposal there is a library that used to monkey patch the Array.prototype with an incompatible groupBy method. When considering new APIs for the web, backwards compatibility is hugely important. This was highligted a few years ago when trying to implement Array.prototype.flatten, in an event known as SmooshGate.

Fortunately, using static methods actually seems better for future extensibility. When the Records and Tuples proposal comes to fruition, we can add a Record.groupBy method for grouping arrays into an immutable record.

JavaScript is filling in the gaps

Grouping items together is clearly an important thing we do as developers. lodash.groupBy is currently downloaded from npm between 1.5 and 2 million times a week. It's great to see JavaScript filling in these gaps and making it easier for us to do our jobs.

For now, go get Chrome 117 and try these new methods out for yourself.

Node.js includes built-in support for .env files

Tue, 05 Sep 2023 00:00:00 GMT

With the recent release of version 20.6.0, Node.js now has built-in support for .env files. You can now load environment variables from a .env file into process.env in your Node.js application completely dependency-free.

Loading an .env file is now as simple as:

node --env-file .env

What is .env?

.env files are used to configure environment variables that will be present within a running application. The idea comes from the Twelve-Factor App methodology, which says to store everything that is likely to vary between deploys (e.g. dev, staging, production) in the environment.

Config should not be a part of your application code and should not be checked-in to version control. Things like API credentials, or other secrets, should be stored separately and loaded in the environment in which they are needed. A .env file lets you manage your config for applications where it isn't practical to set variables in the environment, like your development machine or <abbr title="continous integration">CI</abbr>.

There are libraries in many different languages that support using a .env file to load variables into the environment, they are usually called "dotenv", and the Node.js dotenv is no different. But now, Node.js itself supports this behaviour.

How do you use .env in Node.js?

A .env file looks like this:

PASSWORD=supersecret
API_KEY=84de8263ccad4d3dabba0754e3c68b7a
# .env files can have comments too

By convention you would save this as .env in the root of your project, though you can call it whatever you want.

You can then set the variables in the file as environment variables by starting Node.js with the --env-file flag pointing to your .env file. When loaded, the variables are available as properties of process.env.

$ node --env-file .env
Welcome to Node.js v20.6.0.
Type ".help" for more information.
> console.log(process.env.PASSWORD)
supersecret
undefined
> console.log(process.env.API_KEY)
84de8263ccad4d3dabba0754e3c68b7a
undefined

Supported features

Support right now is fairly basic compared to dotenv. For example:

You cannot currently use multiline values
You cannot use variable expansion

But, the feature is under active development. Since the 20.7.0 release, you can now specify multiple files. The variables from the last file will override any previous files.

node --env-file .env --env-file .env.development

There is more work to be done, and some of these features may be added. You can follow the discussion on GitHub here.

Incorrect features

In the 20.6.0 release, the documentation says, "If the same variable is defined in the environment and in the file, the value from the environment takes precedence." This is the way that all dotenv packages work by default. However, that is not currently true of Node.js's implementation and variables in the .env file will override the environment.

This has been fixed as of version 20.7.0. Variables defined in the environment now take precedence over variables in a .env file.

Benefits to Node.js's implementation

Even though this implementation is missing some features, it has some benefits over using a third-party package. Node.js loads and parses the .env file as it is starting up, so you can include environment variables that configure Node.js itself, like NODE_OPTIONS.

So, you can have an .env file that looks like this:

NODE_OPTIONS="--no-warnings --inspect=127.0.0.1:9229"

Then, when you run node --env-file=.env the process will run without emitting warnings and it will activate the inspector on the IP address 127.0.0.1:9229.

Note: you cannot put NODE_OPTIONS="--env-file .env in your .env file. It is disallowed to avoid inifinite loops.

Node.js just keeps improving

Go try out Node.js version 20.6.0! Version 20 has brought new features, like a stable test runner, mock timers, and now .env file support, as well as many other upgrades, fixes and improvements. Version 20 becomes the active <abbr title="long term support">LTS</abbr> version of Node.js in October, so now is a good time to test these new features out and start considering upgrading your application to take advantage.

Easy and accessible pagination links for your Astro collections

Thu, 13 Jul 2023 00:00:00 GMT

Generating pagination links is not as straightforward as it may seem. So, while rebuilding my own site with Astro, I released a <Pagination /> component on npm as @philnash/astro-pagination that anyone can use in their Astro site. Read on to find out more.

Pagination

Pagination is something that most content sites need. It is often better to list collections with lots of entries, like blog posts, across multiple pages because a single page would be overwhelming to scroll through.

Astro provides the paginate function to the callback to getStaticPaths to make it easy to turn a collection into a number of paths and pages. Once you have turned your collection into a list of pages, you then need to render links that your users can use to navigate through the list.

While this may at first seem straightforward, as with many things on the web, there are hidden depths to it. Not only do you need to write the code that parses the current page and produces links to the previous page, the next page and a window of pages around the current one, you also need to produce accessible HTML that will be easy to use for any of your site's visitors.

An Astro Pagination component

To make this easy for anyone building with Astro, I released @philnash/astro-pagination on npm. It is a <Pagination /> component that you can use in your Astro site to create pagination links based on a Page object.

How to use it

Start by installing the package:

npm install @philnash/astro-pagination

Then in a list page, you can import and use the Pagination component. The component requires two properties, a page and a urlPattern. The page should be an Astro Page object, typically provided through Astro.props. The urlPattern should be the path you are using for your paginated pages, with a {} where the page number should go. A simplified Astro page might look something like this:

---
import Pagination from "@philnash/astro-pagination";

export async function getStaticPaths({ paginate }) { /* ... */ }
const { page } = Astro.props;
---

{ /* render the items from the page */ }

<Pagination page={page} urlPattern="/blog/page/{}" />

This will render HTML that looks like:

<nav role="navigation" aria-label="Pagination">
  <ul>
    <li>
      <a
        href="/blog/page/4"
        class="previous-page"
        aria-label="Go to previous page"
        >« Prev</a
      >
    </li>
    <li>
      <a class="number" href="/blog/page" aria-label="Go to page 1"> 1 </a>
    </li>
    <li>
      <span class="start-ellipsis">…</span>
    </li>
    <li>
      <a class="number" href="/blog/page/3" aria-label="Go to page 3"> 3 </a>
    </li>
    <li>
      <a class="number" href="/blog/page/4" aria-label="Go to page 4"> 4 </a>
    </li>
    <li>
      <em aria-current="page" aria-label="Current page, page 5"> 5 </em>
    </li>
    <li>
      <a class="number" href="/blog/page/6" aria-label="Go to page 6"> 6 </a>
    </li>
    <li>
      <a class="number" href="/blog/page/7" aria-label="Go to page 7"> 7 </a>
    </li>
    <li>
      <span class="end-ellipsis">…</span>
    </li>
    <li>
      <a class="number" href="/blog/page/10" aria-label="Go to page 10"> 10 </a>
    </li>
    <li>
      <a href="/blog/page/6" class="next-page" aria-label="Go to next page"
        >Next »</a
      >
    </li>
  </ul>
</nav>

This renders like this:

Well, it renders like that with my site's CSS applied. You will need to style it yourself.

The generated markup includes a bunch of things, including accessibility features based on research from a11ymatters on accessible pagination. There is:

a link to the previous page
a link to the first page
a window of links around the current page
ellipses to show where pages exist between the first/last page and the current window
a link to the last page
a link to the next page
a <nav> element around the links, with a role attribute set to "navigation" and an aria-label attribute to describe it as "Pagination"
a list of links, allowing assistive technology to announce how many items there are in the list and navigate through them
an aria-label attribute on each link to provide a full description of the link's destination
an aria-current attribute on the element representing the current page
a helpful class name on each of the important elements to allow for styling

Advanced usage

There are more properties you can pass to the <Pagination /> component that give you greater control over the output. They include properties like previousLabel and nextLabel, that lets you set the text for the previous and next links, or windowSize, that lets you determine how many pages are shown in the middle of the range. You can see all the the available options in the documentation.

Future improvements

While the <Pagination /> component is ready to be used, and is already in use on my own site, there are definitely some improvements that I will be adding. For example, you should be able to:

use a component for the previousLabel and nextLabel
style the links like other Astro components
add class names to the elements, so you can style using utility CSS frameworks
handle internationalisation

Ultimately, I'd love for this to be a component that every Astro site considers using for pagination, from my blog right here to the Astro blog itself.

Any feedack?

I tried to make this component simple to use and flexible. As I've described above, there's plenty more to do, but it's always worth asking for feedback too.

So, would you use this component to simplify pagination in your own Astro sites? Is there anything you'd add or change? Let me know on Twitter, another social network of choice, or in the GitHub issues.

Astro Pagination

Check out the source on GitHub (give it a star, if you fancy) and let me know if you use this component.

Build bots on Bluesky with Node.js and GitHub Actions

Tue, 16 May 2023 00:00:00 GMT

Bluesky is the new social network in town and it's an exciting place to explore right now. I was fortunate enough to get an invite early on and take part in the early community. But Bluesky is not just a Twitter clone, it's an application on top of The AT Protocol, a (still being built) federated protocol for social networks with some interesting properties.

Because there's a protocol, that also means there's an API. If you remember far enough back to the early days of Twitter, the API drove a lot of exciting applications and features for users. There was a swarm of activity around Twitter's API and now, even in the early days, there is a similar excitement about Bluesky's API. So let's look into how to get started with the API using TypeScript and build a bot that runs on a schedule using GitHub Actions.

What you will need

In order to build against the Bluesky API you will need an account. As I write this, accounts are invite only, but as the application and protocol stabilises, I expect there to me more available.

You will also need:

Node.js version 22
A GitHub account

Getting started with the API

Let's write a quick script to post a status to Bluesky from JavaScript. In your terminal create a new Node.js project:

mkdir bluesky-bot
cd bluesky-bot
npm init --yes

Add an .nvmrc file with the version 18.16.0 so that we can guarantee the Node.js version this will run on.

echo 22 > .nvmrc

Install the AT Protocol/Bluesky API client:

npm install @atproto/api

This library gives us easy access to the Bluesky API, a rich text library for formatting links and mentions, and lower level access to the AT Protocol itself.

Send your first Bluesky post

Create a file called index.js and open it in your editor.

Start by requiring the AtpAgent class from the package.

const { AtpAgent } = require("@atproto/api");

Create an asynchronous function that will connect to the service and send a post. Within that function instantiate a new agent, setting the service to "https://bsky.social", the only available AT Protocol service at the moment. Then log the agent in with your Bluesky identifier (your username) and an app password. This is just a quick script to get going with, so we're just going to embed our credentials for now, in reality you want to keep credentials out of your code and load them through environment variables or similar.

async function sendPost() {
  const agent = new AtpAgent({ service: "https://bsky.social" });
  await agent.login({
    identifier: "YOUR_IDENTIFIER_HERE",
    password: "YOUR_PASSWORD_HERE",
  });
  // ...
}

Once you've logged in, you can then use the agent to post a status.

async function sendPost(text) {
  const agent = new AtpAgent({ service: "https://bsky.social" });
  await agent.login({
    identifier: "YOUR_IDENTIFIER_HERE",
    password: "YOUR_PASSWORD_HERE",
  });
  await agent.post({ text });
}

Now you can call the sendPost method with the text you want to send to the API:

sendPost("Hello from the Bluesky API!");

Run this code in your terminal with node index.js and you will see your first post from the API on your Bluesky account.

Sending posts with rich text

If you want to send links or mentions on the platform you can't just send plain text. Instead you need to send rich text, and the library provides a function to create that. Let's update the above code to generate rich text and use it to make a post.

First, require the RichText module.

const { AtpAgent, RichText } = require("@atproto/api");

Then take the text you want to send and create a new RichText object with it. Use that object to detect the facets in the text, then pass both the text and the facets to the post method.

async function sendPost(text) {
  const agent = new AtpAgent({ service: "https://bsky.social" });
  await agent.login({
    identifier: "YOUR_IDENTIFIER_HERE",
    password: "YOUR_PASSWORD_HERE",
  });
  const richText = new RichText({ text });
  await richText.detectFacets(agent);
  await agent.post({
    text: richText.text,
    facets: richText.facets,
  });
}

If you call the sendPost function with text that includes a user mention or a link, it will be correctly linked in Bluesky and notify the mentioned user.

sendPost("Hello from the Bluesky API! Hi @philna.sh!");

That's the basics on creating posts using the Bluesky API. Now let's take a look at scheduling the posts.

Scheduling with GitHub Actions

GitHub Actions lets you automate things in your repositories. This means we can use it to automate posting to Bluesky. One of the triggers for a GitHub Action is the schedule which lets you run a workflow at specified times using cron syntax.

We can add a GitHub Actions workflow to this application that will start working when we push the repo up to GitHub. Before we do that, we should remove our credentials first. Update the code that logs in to the Bluesky service to use environment variables instead of hard coding the credentials:

  await agent.login({
    identifier: process.env.BSKY_HANDLE,
    password: process.env.BSKY_PASSWORD,
  });

Next, create a directory called .github with a directory called workflows inside.

mkdir -p .github/workflows

Create a YAML file called post.yml and open it in your editor. Add the following:

name: "Post to Bluesky"

on: workflow_dispatch

jobs:
  post:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Use Node.js
        uses: actions/setup-node@v3
        with:
          node-version-file: ".nvmrc"
      - run: npm ci
      - name: Send post
        run: node index.js
        env:
          BSKY_HANDLE: ${{ secrets.BSKY_HANDLE }}
          BSKY_PASSWORD: ${{ secrets.BSKY_PASSWORD }}

This workflow file does a bunch of things. It sets up one job called post that will:

run on the latest Ubuntu
checkout the repository
install the Node.js version listed in the .nvmrc file we created earlier
install the dependencies
run the index.js file, with two secrets added to the environment

In the workflow above the workflow_dispatch trigger is present instead of the schedule trigger. The workflow_dispatch trigger allows you to start a workflow by visiting it in the GitHub UI and pressing a button. It's a great way to test your workflow is working without having to wait for the schedule or push a new commit.

Create a GitHub repo, and push all this code up to it. In the repo settings, find Actions secrets and variables. Add two secrets called BSKY_HANDLE and BSKY_PASSWORD which contain the credentials you were using earlier.

Testing it out

With your code and secrets in place head to the Actions tab for your repo. Click on the workflow called "Post to Bluesky" and then find the button that says "Run workflow". This is the workflow_dispatch trigger and it will run the workflow, eventually running your code and posting to Bluesky. Use this to test out the workflow and any changes to the code before you eventually write the schedule.

Scheduling the workflow

Once you are happy with your code and that the workflow is working it's time to set up a schedule. Remove the workflow_dispatch trigger and replace it with the schedule trigger, which looks like this:

on:
  schedule:
    - cron: "30 5,17 * * *"

I don't read cron, but crontab.guru tells me that this would run the workflow at 5:30 and 17:30 every day. I recommend playing around with that tool to get your schedule correct.

Once you are happy, save, commit and push to GitHub and your Bluesky bot will set off posting.

A template to make this easier

To make this easier, I created a template that has all of the above ready to go for you. Hit the big green "Use this template" button on the repo and you will get your own project ready to go. All you need to do is provide your own function that will return the text that will get posted to Bluesky. There are also instructions in the README to walk you through it all.

My first bot

I've used this template repo to create my first bot on the Bluesky platform. It's a simple but fun one. It posts an hourly dad joke to Bluesky from the icanhazdadjoke.com API. You can find the code for this bot on GitHub too.

Bluesky is going to be a lot of fun

When Twitter first started the availability of the API caused a wave of creativity from developers. Even though Bluesky remains in very early invite only mode there is already a lot of things being built and it is exciting to see.

I'm looking forward to creating bots with this method, but also exploring more of the API, data and protocol to see what can be achieved.

If you have an account with Bluesky, come follow me here. See you in the sky!

Create a CLI Chatbot with the ChatGPT API and Node.js

Mon, 13 Mar 2023 00:00:00 GMT

ChatGPT has taken the world by storm and this week OpenAI released the ChatGPT API. I've spent some time playing with ChatGPT in the browser, but the best way to really get on board with these new capabilities is to try building something with it. With the API available, now is that time.

This was inspired by Greg Baugues's implementation of a chatbot command line interface (CLI) in 16 lines of Python. I thought I'd start by trying to build the same chatbot but using JavaScript.

(It turns out that Ricky Robinett also had this idea and published his bot code here, it's pleasing to see how similar the implementations are!)

The code

It turns out that Node.js requires a bit more code to deal with command line input than Python, so where Greg's version was 16 lines mine takes 31. Having built this little bot, I'm no less excited about the potential for building with this API though.

Here's the full code, I'll explain what it is doing further down.

import { createInterface } from "node:readline/promises";
import { stdin as input, stdout as output, env } from "node:process";
import { Configuration, OpenAIApi } from "openai";

const configuration = new Configuration({ apiKey: env.OPENAI_API_KEY });
const openai = new OpenAIApi(configuration);
const readline = createInterface({ input, output });

const chatbotType = await readline.question(
  "What type of chatbot would you like to create? "
);
const messages = [{ role: "system", content: chatbotType }];
let userInput = await readline.question("Say hello to your new assistant.\n\n");

while (userInput !== ".exit") {
  messages.push({ role: "user", content: userInput });
  try {
    const response = await openai.createChatCompletion({
      messages,
      model: "gpt-3.5-turbo",
    });

    const botMessage = response.data.choices[0].message;
    if (botMessage) {
      messages.push(botMessage);
      userInput = await readline.question("\n" + botMessage.content + "\n\n");
    } else {
      userInput = await readline.question("\nNo response, try asking again\n");
    }
  } catch (error) {
    console.log(error.message);
    userInput = await readline.question("\nSomething went wrong, try asking again\n");
  }
}

readline.close();

When you run this code it looks like this:

Let's dig into how it works and how you can build your own.

Building a chatbot

You will need an OpenAI platform account to interact with the ChatGPT API. Once you have signed up, create an API key from your account dashboard.

As long as you have Node.js installed, the only other thing you'll need is the openai Node.js module.

Let's start a Node.js project and create this CLI application. First create a directory for the project, change into it and initialise it with npm:

mkdir chatgpt-cli
cd chatgpt-cli
npm init --yes

Install the openai module as a dependency:

npm install openai

Open package.json and add the key "type": "module" to the configuration, so we can build this as an ES module which will allow us to use top level await.

Create a file called index.js and open it in your editor.

Interacting with the OpenAI API

There are two parts to the code, dealing with input and output on the command line and dealing with the OpenAI API. Let's start by looking at how the API works.

First we import two objects from the openai module, the Configuration and OpenAIApi. The Configuration class will be used to create a configuration that holds the API key, you can then use that configuration to create an OpenAIApi client.

import { env } from "node:process";
import { Configuration, OpenAIApi } from "openai";

const configuration = new Configuration({ apiKey: env.OPENAI_API_KEY });
const openai = new OpenAIApi(configuration);

In this case, we'll store the API key in the environment and read it with env.OPENAI_API_KEY.

To interact with the API we now use the OpenAI client to create chat completions for us. OpenAI's text generating models don't actually converse with you, but are built to take input and come up with plausible sounding text that would follow that input, a completion. With ChatGPT, the model is configured to receive a list of messages and then come up with a completion for the conversation. Messages in this system can come from one of 3 different entities, the "system", "user" and "assistant". The "assistant" is ChatGPT itself, the "user" is the person interacting and the system allows the program (or the user, as we'll see in this example) to provide instructions that define how the assistant behaves. Changing the system prompts for how the assistant behaves is one of the most interesting things to play around with and allows you to create different types of assistants.

With our openai object configured as above, we can create messages to send to an assistant and request a response like this:

const messages = [
  { role: "system", content: "You are a helpful assistant" },
  { role: "user", content: "Can you suggest somewhere to eat in the centre of London?" }
];
const response = await openai.createChatCompletion({
  messages,
  model: "gpt-3.5-turbo",
});
console.log(response.data.choices[0].message);
// => "Of course! London is known for its diverse and delicious food scene..."

As the conversation goes on, we can add the user's questions and assistant's responses to the messages array, which we send with each request. That gives the bot history of the conversation, context for which it can build further answers on.

To create the CLI, we just need to hook this up to user input in the terminal.

Interacting with the terminal

Node.js provides the Readline module which makes it easy to receive input and write output to streams. To work with the terminal, those streams will be stdin and stdout.

We can import stdin and stdout from the node:process module, renaming them to input and output to make them easier to use with Readline. We also import the createInterface function from node:readline

import { createInterface } from "node:readline/promises";
import { stdin as input, stdout as output } from "node:process";

We then pass the input and output streams to createInterface and that gives us an object we can use to write to the output and read from the input, all with the question function:

const readline = createInterface({ input, output });

const chatbotType = await readline.question(
  "What type of chatbot would you like to create? "
);

The above code hooks up the input and output stream. The readline object is then used to post the question to the output and return a promise. When the user replies by writing into the terminal and pressing return, the promise resolves with the text that the user wrote.

Completing the CLI

With both of those parts, we can write all of the code. Create a new file called index.js and enter the code below.

We start with the imports we described above:

import { createInterface } from "node:readline/promises";
import { stdin as input, stdout as output, env } from "node:process";
import { Configuration, OpenAIApi } from "openai";

Then we initialise the API client and the Readline module:

const configuration = new Configuration({ apiKey: env.OPENAI_API_KEY });
const openai = new OpenAIApi(configuration);
const readline = createInterface({ input, output });

Next, we ask the first question of the user: "What type of chatbot would you like to create?". We will use the answer of this to create a "service" message in a new array of messages that we will continue to add to as the conversation goes on.

const chatbotType = await readline.question(
  "What type of chatbot would you like to create? "
);
const messages = [{ role: "system", content: chatbotType }];

We then prompt the user to start interacting with the chatbot and start a loop that says while the user input is not equal to the string ".exit" keep sending that input to the API. If the user enters ".exit" the program will end, like in the Node.js REPL.

let userInput = await readline.question("Say hello to your new assistant.\n\n");

while (userInput !== ".exit") {
  // loop
}

readline.close();

Inside the loop we add the userInput to the messages array as a "user" message. Then, within a try/catch block, send it to the OpenAI API. We set the model as "gpt-3.5-turbo" which is the underlying name for ChatGPT.

When we get a response from the API we get the message out of the response.data.choices array. If there is a message we store it as an "assistant" message in the array of messages and output it to the user, waiting for their input again using readline. If there is no message in the response from the API, we alert the user and wait for further user input. Finally, if there is an error making a request to the API we catch the error, log the message and tell the user to try again.

while (userInput !== ".exit") {
  messages.push({ role: "user", content: userInput });
  try {
    const response = await openai.createChatCompletion({
      messages,
      model: "gpt-3.5-turbo",
    });

    const botMessage = response.data.choices[0].message;
    if (botMessage) {
      messages.push(botMessage);
      userInput = await readline.question("\n" + botMessage.content + "\n\n");
    } else {
      userInput = await readline.question("\nNo response, try asking again\n");
    }
  } catch (error) {
    console.log(error.message);
    userInput = await readline.question(
      "\nSomething went wrong, try asking again\n"
    );
  }
}

Put that all together and you have your assistant. The full code is at the top of this post or on GitHub.

You can now run the assistant by passing it your OpenAI API key as an environment on the command line:

OPENAI_API_KEY=YOUR_API_KEY node index.js

This will start your interaction with the assistant, starting with it asking what kind of assistant you want. Once you've declared that, you can start chatting with it.

Experimenting helps us to understand

Personally, I'm not actually sure how useful ChatGPT is. It is clearly impressive, its ability to return text that reads as if it was written by a human is incredible. However, it returns content that is not necessarily correct, regardless of how confidently it presents that content.

Experimenting with ChatGPT is the only way that we can try to understand what it useful for, thus building a simple chat bot likes this gives us grounds for that experiment. Learning that the system commands can give the bot different personalities and make it respond in different ways is very interesting.

You might have heard, for example, that you can ask ChatGPT to help you with programming, but you could also specify a JSON structure and effectively use it as an API as well. But as you experiment with that you will likely find that it should not be an information API, but more likely something you can use to understand your natural text and turn it into a JSON object. To me this is exciting as it means that ChatGPT could help create more natural voice assistants, that can translate meaning from speech better than the existing crop that expect commands to be given in a more exact manner. I still have experimenting to do with this idea, and having this tool gives me that opportunity.

This is just the beginning

If experimenting with this technology is the important thing for us to understand what we can build with it and what we should or should not build with it, then making it easier to experiment is the next goal. My next goal is to expand this tool so that it can save, interact with and edit multiple assistants so that you can continue to work with them and improve them over time.

In the meantime, you can check out the full code for this first assistant in GitHub, follow the repo to keep up with improvements.

The yaml document from hell — JavaScript edition

Thu, 02 Feb 2023 00:00:00 GMT

I recently came across this blog post from Ruud van Asseldonk titled "The yaml document from hell". I've always heard that yaml has its pitfalls, but hadn't looked into the details and thankfully hadn't been affected, mainly due to my very infrequent and simple use of yaml. If you are in the same boat as me, I recommend reading that article now as I almost can't believe I've avoided any issues with it.

The article digs into the issues in the yaml spec itself, and then describes what happens in Python's PyYAML and Golang's yaml library with an example file, the titular yaml document from hell. I wanted to see how things were in the JavaScript ecosystem.

Yaml in JavaScript

A search for JavaScript yaml parsers on npm brings up yaml (which I have used in my own project) and js-yaml. js-yaml has the most weekly downloads according to npm and the most stars on GitHub however yaml seems to be under more active development, having been most recently published (a month ago at the time of writing) compared to js-yaml's last publish date almost 2 years ago. There is also yamljs, but the project hasn't received a commit since November 2019 and hasn't been released for 6 years, so I am going to disregard it for now.

Let's see what yaml and js-yaml do with the yaml document from hell.

The document itself

To save yourself from going back and forth between van Asseldonk's article and this one, here is the yaml document.

server_config:
  port_mapping:
    # Expose only ssh and http to the public internet.
    - 22:22
    - 80:80
    - 443:443

  serve:
    - /robots.txt
    - /favicon.ico
    - *.html
    - *.png
    - !.git  # Do not expose our Git repository to the entire world.

  geoblock_regions:
    # The legal team has not approved distribution in the Nordics yet.
    - dk
    - fi
    - is
    - no
    - se

  flush_cache:
    on: [push, memory_pressure]
    priority: background

  allow_postgres_versions:
    - 9.5.25
    - 9.6.24
    - 10.23
    - 12.13

So how do our JavaScript libraries handle this file?

The failures

Anchors, aliases, and tags

Let's start with the failures. As described in the original article under the subhead "Anchors, aliases, and tags" this section is invalid:

  serve:
    - /robots.txt
    - /favicon.ico
    - *.html
    - *.png
    - !.git  # Do not expose our Git repository to the entire world.

This causes both of our JavaScript yaml libraries to throw an error, both referencing an undefined alias. This is because the * is a way to reference an anchor created earlier in the document using an &. In our document's case, that anchor was never created, so this is a parsing error.

If you want to learn more about anchors and aliases it seems like something that is important in build pipelines. Both Bitbucket and GitLab have written about how to use anchors to avoid repeating sections in yaml files.

For the purposes of trying to get the file to parse, we can make those aliases strings as they were likely intended.

  serve:
    - /robots.txt
    - /favicon.ico
    - "*.html"
    - "*.png"
    - !.git  # Do not expose our Git repository to the entire world.

Now we get another parsing error from our libraries; both of them complain about an unknown or unresolved tag. The ! at the start of !.git is the character triggering this behaviour.

Tags seem to be the most complicated part of yaml to me. They depend on the parser you are using and allow that parser to do something custom with the content that follows the tag. My understanding is that you could use this in JavaScript to, say, tag some content to be parsed into a Map instead of an Object or a Set instead of an Array. Van Asseldonk explains this with this alarming sentence:

This means that loading an untrusted yaml document is generally unsafe, as it may lead to arbitrary code execution.

PyYaml apparently has a safe_load method that will avoid this, but Go's yaml package doesn't. It seems that the JavaScript libraries also lack this feature, so the warning for untrusted yaml documents stands.

If you do want to take advantage of the tag feature in yaml, you can check out the yaml package's documentation on custom data types or js-yaml's supported yaml types and unsafe type extensions.

To make the yaml file parse, let's encase all the weird yaml artifacts in quotes to make them strings:

  serve:
    - /robots.txt
    - /favicon.ico
    - "*.html"
    - "*.png"
    - "!.git"  # Do not expose our Git repository to the entire world.

With the serve block looking it does above, the file now parses. So what happens to the rest of the potential yaml gotchas?

Accidental numbers

One thing that I am gathering from this investigation so far is that if you need something to be a string, do not be ambiguous about it, surround it in quotes. That counted for the aliases and tags above and it also counts for accidental numbers. In the following section of the yaml file you see a list of version numbers:

  allow_postgres_versions:
    - 9.5.25
    - 9.6.24
    - 10.23
    - 12.13

Version numbers are strings, numbers can't have more than one decimal point in them. But when this is parsed by either JavaScript library the result is as follows:

  allow_postgres_versions: [ '9.5.25', '9.6.24', 10.23, 12.13 ]

Now we have an array of strings and numbers. If a yaml parser thinks something looks like a number it will parse it as such. And when you come to use those values they might not act as you expect.

Version numbers in GitHub Actions

I have had this issue within GitHub Actions before. It was in a Ruby project, but this applies to anyone trying to use version numbers in a GitHub Actions yaml file. I tried to use a list of Ruby version numbers, this worked fine up until Ruby version 3.1 was released. I had 3.0 in the array. Within GitHub Actions this was parsed as the integer 3. This might seem fine, except that when you give an integer version to GitHub Actions it picks the latest minor point for that version. So, once Ruby 3.1 was released, the number 3.0 would select version 3.1. I had to make the version number a string, "3.0", and then it was applied correctly.

Accidental numbers cause issues. If you need a string, make sure you provide a string.

The successes

It's not all bad in the JavaScript world. After working through the issues above, we might now be in the clear. Let's take a look now at what parsed correctly from this yaml file.

Sexagesimal numbers

Under the port mapping section of the yaml file we see:

  port_mapping:
    # Expose only ssh and http to the public internet.
    - 22:22
    - 80:80
    - 443:443

That 22:22 is dangerous in yaml version 1.1 and PyYaml parses it as a sexagesimal (base 60) number, giving the result of 1342. Thankfully both JavaScript libraries have implemented yaml 1.2 and 22:22 is parsed correctly as a string in this case.

  port_mapping: [ '22:22', '80:80', '443:443' ]

The Norway problem

In yaml 1.1 no is parsed as false. This is known as "the Norway problem" because listing countries as two character identifiers is fairly common and having this yaml:

  geoblock_regions:
    - dk
    - fi
    - is
    - no
    - se

parsed into this JavaScript:

  geoblock_regions: [ 'dk', 'fi', 'is', false, 'se' ]

is just not helpful. The good news is that, unlike Go's yaml library, both JavaScript libraries have implemented yaml 1.2 and dropped no as an alternative for false. The geoblock_regions sections is successfully parsed as follows:

  geoblock_regions: [ 'dk', 'fi', 'is', 'no', 'se' ]

Non-string keys

You might believe that keys in yaml would be parsed as strings, like JSON. However they can be any value. Once again there are values that may trip you up. Much like with the Norway problem in which yes and no can be parsed as true and false, the same goes for on and off. This is manifested in our yaml file in the flush_cache section:

  flush_cache:
    on: [push, memory_pressure]
    priority: background

Here the key is on, but in some libraries it is parsed as a boolean. In Python, even more confusingly the boolean is then stringified and appears as the key "True". Thankfully this is handled by the JavaScript libraries and on becomes the key "on".

  flush_cache: { on: [ 'push', 'memory_pressure' ], priority: 'background' }

This is of particular concern in GitHub Actions again, where on is used to determine what events should trigger an Action. I wonder if GitHub had to work around this when implementing their parsing.

Parsing as yaml version 1.1

Many of the issues that our JavaScript libraries sidestep are problems from yaml 1.1 and both libraries have fully implemented yaml 1.2. If you do wish to throw caution to the wind, or you have to parse a yaml file explicitly with yaml 1.1 settings, the yaml library can do that for you. You can pass a second argument to the parse function to tell it to use version 1.1, like so:

import { parse } from "yaml";
const yaml = parse(yamlContents, { version: "1.1" });
console.log(yaml);

Now you get a result with all of the fun described above:

{
  server_config: {
    port_mapping: [ 1342, '80:80', '443:443' ],
    serve: [ '/robots.txt', '/favicon.ico', '*.html', '*.png', '!.git' ],
    geoblock_regions: [ 'dk', 'fi', 'is', false, 'se' ],
    flush_cache: { true: [ 'push', 'memory_pressure' ], priority: 'background' },
    allow_postgres_versions: [ '9.5.25', '9.6.24', 10.23, 12.13 ]
  }
}

Note that in this case I left the aliases and tags quoted as strings so that the file could be parsed successfully.

Stick with version 1.2, the default in both JavaScript yaml libraries, and you'll get a much more sensible result.

Isn't yaml fun?

In this post we've seen that it's easy to write malformed yaml if you weren't aware of aliases or tags. It's also easy to write mixed arrays of strings and numbers. There are also languages and libraries in which yaml 1.1 is still hanging around and on. yes, off, and no are booleans and some numbers can be parsed into base 60.

My advice, after going through all of this, is to err on the side of caution when writing yaml. If you want a key or a value to be a string, surround it in quotes and explicitly make it a string.

On the other hand, if you are parsing someone else's yaml then you will need to program defensively and try to handle the edge cases, like accidental numbers, that can still cause issues.

Finally, if you have the option, choose a different format to yaml. Yaml is supposed to be human-friendly, but the surprises and the bugs that it can produce are certainly not developer-friendly and ultimately that defeats the purpose.

The conclusion to the original yaml document from hell post suggests many alternatives to yaml that will work better. I can't help but think that in the world of JavaScript that something JSON based, but friendlier to author, should be the solution.

There is a package that simply strips comments from JSON or there's JSON5 a JSON format that aims to be easier to write and maintain by hand. JSON5 supports comments as well as trailing commas, multiline strings, and various number formats. Either of these are a good start if you want to make authoring JSON easier and parsing hand authored files more consistent.

If you can avoid yaml, I recommend it. If you can't, good luck.

Better two factor authentication experiences with WebOTP

Wed, 07 Dec 2022 00:00:00 GMT

Two factor authentication (2FA) is a great way to improve the security of user accounts in an application. It helps protect against common issues with passwords, like users picking easily guessable passwords or reusing the same password across multiple sites. There are different ways to implement two factor authentication, including SMS, using an authenticator application and WebAuthn.

SMS is the most widely used and won't be going away, so it falls on us as developers to do our best to build the best SMS 2FA experience for our users. The WebOTP API is one way we can help reduce friction in the login experience and even provide some protection against phishing.

What is the WebOTP API?

The WebOTP API is an extension to the Credential Management API. The Credential Management API started by giving us the ability to store and access credentials in a browser's password manager, but now encompasses WebAuthn and two factor authentication. The WebOTP API allows us to request permission from the user to read a 2FA code out of an incoming SMS message.

When you implement the WebOTP API the second step of a login process can go from an awkward process of reading and copying a number of digits from an SMS, to a single button press. A great improvement, I think you'll agree.

How does it work?

To implement WebOTP you will need to do two things:

Update the message you send with the WebOTP format
Add some JavaScript to the login page to request permission to read the message

The SMS message

To have the WebOTP API recognise a message as an incoming 2FA code you need to add a line to the end of the message that you send. That line must include an @ symbol followed by the domain for the site that your user will be logging in to, then a space, the # symbol and then the code itself. If your user is logging in on example.com and the code you are sending them is 123456 then the message needs to look like this:

Your code to log in to the application is 123456

@example.com #123456

The domain ties the message to the website the user should be logging onto. This helps protect against phishing, WebOTP can't be used to request the code from an SMS if the domain the user is logging in to doesn't match the domain in the message. Obviously it can't stop a user copying a code across from a message, but it might give them pause if they come to expect this behaviour.

The JavaScript

Once you have your messages set up in the right format you need some JavaScript on your 2nd factor page that will trigger the WebOTP API, ask the user permission for access to the message and collect the code.

The most minimal version of this code looks like this:

if ('OTPCredential' in window) {
  navigator.credentials.get({
    otp: {
      transport: ['sms']
    }
  }).then((otp) => {
    submitOTP(otp.code);
  });
}

We ask the navigator.credentials object to get a one time password (OTP) from the SMS transport. If the browser detects an incoming message with the right domain and a code in it, the user will be prompted, asking for access. If the user approves the promise resolves with an otp object which has a code property. You can then submit that code to the form and complete the user's login process.

A more complete version of the code, that handles things like finding an input and form, cancelling the request if the form is submitted, and submitting the form if the request is successful, looks like this:

if ('OTPCredential' in window) {
  window.addEventListener('DOMContentLoaded', e => {
    const input = document.querySelector('input[autocomplete="one-time-code"]');
    if (!input) return;
    const ac = new AbortController();
    const form = input.closest('form');
    if (form) {
      form.addEventListener('submit', e => ac.abort());
    }
    navigator.credentials.get({
      otp: { transport:['sms'] },
      signal: ac.signal
    }).then(otp => {
      input.value = otp.code;
      if (form) {
        form.submit();
      }
    }).catch(err => {
      console.error(err);
    });
  });
}

This will work for many sites, but copying and pasting code isn't the best way to share code, so I came up with something a bit easier.

Declarative WebOTP with web components

On Safari, you can get similar behaviour to the WebOTP API by adding one attribute to the <input> element for the OTP code. Setting autocomplete="one-time-code" will trigger Safari to offer the code from the SMS via autocomplete.

Inspired by this, I wanted to make WebOTP just as easy. So, I published a web component, the <web-otp-input> component, that handles the entire process. You can see all the code and how to use it on GitHub. For a quick example, you can add the component to your page as an ES module:

<script type="module" src="https://unpkg.com/@philnash/web-otp-input"></script>

Or install it to your project from npm:

npm install @philnash/web-otp-input

and import it to your application:

import { WebOTPInput } from "@philnash/web-otp-input";

You can then wrap the <web-otp-input> around your existing <input> within a <form>, like this:

<form action="/verification" method="POST">
  <div>
    <label for="otp">Enter your code:</label>
    <web-otp-input>
      <input type="text" autocomplete="one-time-code" inputmode="numeric" id="otp" name="otp" />
    </web-otp-input>
  </div>
  <button type="submit">Submit</button>
</form>

Then the WebOTP experience will happen automatically for anyone on a browser that supports it, without writing any additional JavaScript.

WebOTP: a better experience

The WebOTP API makes two factor authentication with SMS a better experience. For browsers that support it, entering the code that is sent as a second factor becomes a breeze for users.

There are even circumstances where it works for desktop browsers too. For a user with Chrome on the desktop and Chrome on Android and signed into their Google account on both, signing in on the desktop will cause a notification on the mobile device asking to approve sending the code to the desktop. Approving that on the mobile devices transfers the code to the desktop browser. You don't even have to write more code to handle this, all you need is the JavaScript in this article.

For more on WebOTP, check out these articles:

If you are building two factor authentication or phone verification, consider implementing the WebOTP API as well to make that process easier for your users.

Alias your Mastodon username to your own domain with Jekyll

Wed, 23 Nov 2022 00:00:00 GMT

Mastodon is different to most online services. It is a federated network, so when you set up an account you need to choose a server to use. Your username then becomes a combination of your handle and that server you signed up to. For example, I am currently @philnash@mastodon.social.

But what if you want to personalise that a bit more? What if you wanted to use your own domain for your Mastodon account without having to host a whole Mastodon server? Using your own domain means that no matter what instance you used, or if you moved instance, you could share one Mastodon username that always pointed to the right profile and was personalised to your own site.

WebFinger to the rescue

It turns out that you can do this. Maarten Balliauw wrote about how Mastodon uses WebFinger to attach extra information to an email address. Information like an associated profile page or ActivityPub stream.

Implementing WebFinger requires your domain to respond to a request to /.well-known/webfinger with a JSON representation of the associated accounts. If you have a Mastodon account you can check out what your WebFinger JSON looks like by making a request to https://#{instance}/.well-known/webfinger?resource=acct:#{username}@#{instance}. For example, my WebFinger JSON is available at this URL: https://mastodon.social/.well-known/webfinger?resource=acct:philnash@mastodon.social.

To associate a Mastodon account with your own domain, you can serve this JSON yourself from a /.well-known/webfinger endpoint.

WebFinger with Jekyll

As Maarten pointed out in his post, you can copy the JSON response from your Mastodon instance to a file that you then serve from your own site. My site is powered by Jekyll, so I wanted to make it easy for me, and anyone else using Jekyll, to create and serve that WebFinger JSON. I've also built Jekyll plugins before, like jekyll-gzip, jekyll-brotli, jekyll-zopfli, and jekyll-web_monetization.

I got to work and built jekyll-mastodon_webfinger.

How to use it

You can serve up your own WebFinger JSON on your Jekyll site to point to your Mastodon profile by following these steps:

Add jekyll-mastodon_webfinger to your Gemfile:
```
bundle add jekyll-mastodon_webfinger
```
Add the plugin to your list of plugins in _config.yml:
```
plugins:
  - jekyll/mastodon_webfinger
```

Add your Mastodon username and instance to _config.yml:

mastodon:
  username: philnash
  instance: mastodon.social

Next time you build the site, you will find a /.well-known/webfinger file in your output directory, and when you deploy you will be able to refer to your Mastodon account using your own domain.

You can see the result of this by checking the WebFinger endpoint on my domain: https://philna.sh/.well-known/webfinger or by searching for @phil@philna.sh on your Mastodon instance.

As this is a static file it sort of acts like a catch-all email address. You can actually search for @any_username@philna.sh and you will find me. If you wanted to restrict this, you would need to build an endpoint that could respond dynamically to the request.

Other ways to serve Mastodon WebFinger responses

I'm not the only one to have considered this. Along with Maarten's original post on the topic, others have built tools or posted about how to do this with your own site.

Lindsay Wardell wrote up how to integrate Mastodon with Astro including showing how to display her feed within her Astro site.

Dominik Kundel put together a Netlify plugin that generates a Mastodon WebFinger file for your Netlify hosted site.

Take a trip into the Fediverse

An interesting side-effect of the increase in popularity of Mastodon is learning and understanding the protocols that underpin federating a social network like this. WebFinger and ActivityPub are having their moment and I look forward to see what further integrations and applications can be built on top of them.

In the meantime, you can use the techniques in this post to use your own domain as an alias for your Mastodon profile. And if you fancy it, connect with me on Mastodon by searching for @phil@philna.sh or at https://mastodon.social/@philnash.

How to use the Bitly API in Ruby

Wed, 09 Nov 2022 00:00:00 GMT

Link shortening has been around for a long time and Bitly is arguably the king of the link shorteners. It has support for shortening long URLs as well as custom short links, custom domains, and metrics to track how each link is performing.

For those of us with the power of code at our fingertips, Bitly also has an API. With the Bitly API you can build all of the functionality of Bitly into your own applications and expose it to your users. In this post you'll learn how to use the Bitly Ruby gem to use the Bitly API in your Ruby applications.

Getting started

To start shortening or expanding links with the Bitly gem, you'll need Ruby installed and a Bitly account.

To make API requests against the Bitly API you will need an access token. Log in to your Bitly account and head to the API settings. Here you can enter your account password and generate a new token. This token will only be shown once, so copy it now.

Using the Bitly API

Open a terminal and install the Bitly gem:

gem install bitly

Let's explore the gem in the terminal. Open an irb session:

irb

Require the gem:

require "bitly"

Create an authenticated API client using the token you created in your Bitly account:

client = Bitly::API::Client.new(
  token: "Enter your access token here"
)

Shortening a URL

You can now use this client object to access all the Bitly APIs. For example, you can shorten a URL to a Bitlink like this:

long_url = "https://twitter.com/philnash"
bitlink = client.shorten(long_url: long_url)
bitlink.link
# => "https://bit.ly/3zYdN21"

The shorten endpoint is a simplified method of shortening a URL. You can also use the create endpoint and set other attributes, like adding a title, tags or deeplinks into native applications.

long_url = "https://twitter.com/philnash"
bitlink = client.create_bitlink(
  long_url: long_url,
  title: "Phil Nash on Twitter",
  tags: ["social media", "worth following"]
)
bitlink.link
# => "https://bit.ly/3zYdN21"
bitlink.title
# => "Phil Nash on Twitter"

Expanding a URL

The API client can also be used to expand Bitlinks. You can use the expand method with any Bitlink, not just ones that you have shortened yourself. When you expand a URL you will get back publicly available information about the URL.

bitlink = client.expand(bitlink: "bit.ly/3zYdN21")
bitlink.long_url
# => "https://twitter.com/philnash"
bitlink.title
# => nil
# (title is not public information)

If the URL was a Bitlink from your own account you get more detailed information when you use the bitlink method.

bitlink = client.bitlink(bitlink: "bit.ly/3zYdN21")
bitlink.long_url
# => "https://twitter.com/philnash"
bitlink.title
# => "Phil Nash on Twitter"

Other Bitlink methods

Once you have a bitlink object, you can call other methods on it. If you wanted to update the information about the link, for example, you can use the update method:

bitlink.update(title: "Phil Nash on Twitter. Go follow him")
bitlink.title
# => "Phil Nash on Twitter. Go follow him"

You can also fetch metrics for your link, including the clicks_summary, link_clicks and click_metrics_by_country. For example:

click_summary = bitlink.clicks_summary
click_summary.total_clicks
# => 1
# (not very popular yet)

Methods that return a list of metrics implement Enumerable so you can loop through them using each:

country_clicks = bitlink.click_metrics_by_country
country_clicks.each do |metric|
  puts "#{metric.value}: #{metric.clicks}"
end
# => AU: 1
# (it was just me clicking it)

With these methods, you can create or fetch short links and then retrieve metrics about them. Your application can shorten links and measure their impact with a few lines of code.

There's more!

For advanced uses, you can also authorise other Bitly accounts to create and fetch short links via OAuth2 as well as manage your account's users, groups, and organisations.

A useful little tool

To find out more about using the Bitly API with Ruby, you can read the Bitly API documentation, the Bitly gem's generated documentation and the docs in the GitHub repo.

If you are interested, you can also read a bit more about the backstory of the Bitly Ruby gem. I've been working on this project since 2009, would you believe?

Are you using the Bitly API or do you have any feedback or feature requests? Let me know on Twitter at @philnash or open an issue in the repo.

Don't ever write your own function to parse URL parameters

Sun, 11 Apr 2021 00:00:00 GMT

Sometimes the platform we are building on provides more functionality than we can keep in our own heads. However, depending on the problem, we often find ourselves trying to write the code to solve the issue rather than finding and using the existing solution provided by the platform. I almost fell for this recently when trying to parse a query string.

I can do it myself

A colleagues had a query string they needed to parse in a function, and asked for recommendations on how to do so. For some reason I decided to roll up my sleeves and code directly into Slack. I came up with something like this:

function parse(input) {
  return input
    .split("&")
    .map((pairs) => pairs.split("="))
    .reduce((acc, [k, v]) => {
      acc[k] = decodeURIComponent(v);
      return acc;
    }, {});
}

Looks pretty good, right? It even uses reduce which makes all array operations look extra fancy. And it works:

parse("name=Phil&hello=world");
// => { name: 'Phil', hello: 'world' }

Except for when it doesn't work. Like if the query string uses the + character to encode spaces.

parse("name=Phil+Nash");
// => { name: 'Phil+Nash' }

Or if you have more than one value for a key, like this:

parse("name=Phil&name=John")
// => { name: "John" }

I could have worked to fix these issues, but that would just bring up more questions. Like how should multiple values be represented? Always as an array? On as an array if there is more than one value?

The problem with all of this is that even after thinking about these extra issues, there are likely more hiding out there. All this thinking and coding is a waste anyway, because we have URLSearchParams.

URLSearchParams has a specification and several implementations

The URLSearchParams class is specified in the URL standard and started appearing in browsers in 2016 and in Node.js in 2017 (in version 7.5.0 as part of the URL standard library module and then in version 10.0.0 in the global namespace).

It handles all of the parsing problems above:

const params = new URLSearchParams("name=Phil&hello=world");
params.get("name");
// => "Phil"
params.get("hello");
// => world

const multiParams = new URLSearchParams("name=Phil&name=John");
multiParams.get("name");
// => "Phil"
// ???
multiParams.getAll("name");
// => [ 'Phil', 'John' ]

And it handles more, like iterating over the parameters:

for (const [key, value] of params) {
  console.log(`${key}: ${value}`)
}
// => name: Phil
// => hello: world

for (const [key, value] of multiParams) {
  console.log(`${key}: ${value}`)
}
// => name: Phil
// => name: John

Or adding to the parameters and serialising back to a query string:

multiParams.append("name", "Terry");
multiParams.append("favouriteColour", "red");
multiParams.toString();
// => 'name=Phil&name=John&name=Terry&favouriteColour=red'

The final thing I like about URLSearchParams is that it is available in both the browser and Node.js. Node.js has had the querystring module since version 0.10.0, but when APIs like this are available on the client and server side, then JavaScript developers can be more productive regardless of the environment in which they are working.

As an aside, one of the things I appreciate about the Deno project is their aim for Deno to use web platform APIs where possible.

Use the platform that is available

This post started as a story about choosing to write code to solve a problem that the platform already had solved. Once I realised my mistake I jumped straight back into Slack to correct myself and recommend URLSearchParams. When you understand the capabilities of the platform you are working with you can both code more efficiently and avoid bugs.

I never have to write code to parse URL parameters, I wrote this post to remind myself of that. You never have to either.

Restart your app and not your tunnel with ngrok and nodemon

Mon, 15 Mar 2021 00:00:00 GMT

When I am developing web applications in Node.js, I like the server to restart when I make changes, so I use nodemon. When I am developing an application that consumes webhooks or that I want to share publicly, I use ngrok. In fact, I like ngrok so much, I volunteered to help maintain the Node.js wrapper for ngrok.

Now, you can run ngrok and nodemon separately and things work fine. But what if you always want to run them together and you want just one command to do that. Since nodemon is a Node package and ngrok has a Node wrapper, we can do this. Here's how.

An example with a basic Express app

You might already have an application you want to do with this, but for the purposes of the post, let's create an example Express application. We can create a basic application with the Express generator like this:

npx express-generator test-app
cd test-app
npm install

Start the application with:

npm start

Open your browswer to localhost:3000 and you will see the welcome page. You can also open localhost:3000/users and it will say "respond with a resource". Open routes/users.js and change the route from:

router.get('/', function(req, res, next) {
  res.send('respond with a resource');
});

to:

router.get('/', function(req, res, next) {
  res.send('respond with a list of users');
});

Refresh localhost:3000/users and you will see it still returns "respond with a resource". Stop the application, restart it with npm start, and when you reload the page it will have changed. We can make this better with nodemon.

Starting the app with nodemon

To start the application with nodemon, you first need to install the package:

npm install nodemon --save-dev

You can then run the application by opening package.json and changing the start script from "start": "node ./bin/www" to "start": "nodemon ./bin/www". This works great and your application now restarts when you make changes.

Adding ngrok to the mix

This all works great with nodemon on its own, but now we want to give the application a public URL while we are developing it. We can use ngrok for this and we can build it in using the ngrok Node package. Start by installing ngrok:

npm install ngrok --save-dev

Now, we could add ngrok to the ./bin/www script that the Express generator created for us. But if you do this, then every time you change something, nodemon will restart the application and your ngrok tunnel. If you're using a free or unregistered ngrok account then your ngrok URL will keep changing on every restart. Instead, let's build a script that starts an ngrok tunnel and then uses nodemon to run the application script ./bin/www.

Create a new file in the bin directoy called ./bin/dev. You might need to make this file executable with chmod 755 ./bin/dev. Open it in your editor.

Start by adding a shebang for Node. We'll also add in a protection to make sure this script isn't run in production.

#!/usr/bin/env node

if (process.env.NODE_ENV === "production") {
  console.error(
    "Do not use nodemon in production, run bin/www directly instead."
  );
  process.exitCode = 1;
  return;
}

In this case, if the environment variable NODE_ENV is set to production the script will just return early.

Next, require the ngrok and nodemon packages.

const ngrok = require("ngrok");
const nodemon = require("nodemon");

Use ngrok to open up a tunnel connecting to a port 3000 on localhost, the port that Express uses. To open the tunnel we call on the connect method of ngrok which returns a promise that resolves with the URL of the ngrok tunnel.

ngrok
  .connect({
    proto: "http",
    addr: "3000",
  })
  .then(url => {
    console.log(url);
  })

You can run this in the terminal with ./bin/dev, you will see an ngrok URL logged. So now we have started the ngrok tunnel, but we aren't yet running the Node application.

Let's make the logging a bit nicer and then move on to starting the application with nodemon.

ngrok
  .connect({
    proto: "http",
    addr: "3000",
  })
  .then(url => {
    console.log(`ngrok tunnel opened at: ${url}`);
    console.log("Open the ngrok dashboard at: https://localhost:4040\n");

    nodemon({
      script: "./bin/www",
      exec: `NGROK_URL=${url} node`,
    });
  })

Here we call on nodemon and pass two arguments via an object. The script is the file we want to run to start the application, in this case ./bin/www. The exec option tells nodemon what script to execute to run the script. We set the NGROK_URL environment variable to the URL that ngrok created for us so that we can refer to the ngrok URL within the application if we need it. Then the rest of the exec command is node.

Start the application with ./bin/dev and you will see the application start up. You can load it at localhost:3000 or at the ngrok URL that is logged. You will also find that if you change the response in routes/users.js then it will update on the next refresh. Now you have ngrok and nodemon working together.

Finessing the script

This is working now, but there are a couple more things we can do to improve the script. We can listen to events on nodemon to give us more information about what is happening to the application and when the underlying application quits, we should close the ngrok tunnel too. We should also catch any errors that might happen to ngrok when connecting a tunnel. Here's the full script:

#!/usr/bin/env node

if (process.env.NODE_ENV === "production") {
  console.error(
    "Do not use nodemon in production, run bin/www directly instead."
  );
  process.exitCode = 1;
  return;
}

const ngrok = require("ngrok");
const nodemon = require("nodemon");

ngrok
  .connect({
    proto: "http",
    addr: "3000",
  })
  .then((url) => {
    console.log(`ngrok tunnel opened at: ${url}`);
    console.log("Open the ngrok dashboard at: https://localhost:4040\n");

    nodemon({
      script: "./bin/www",
      exec: `NGROK_URL=${url} node`,
    }).on("start", () => {
      console.log("The application has started");
    }).on("restart", files => {
      console.group("Application restarted due to:")
      files.forEach(file => console.log(file));
      console.groupEnd();
    }).on("quit", () => {
      console.log("The application has quit, closing ngrok tunnel");
      ngrok.kill().then(() => process.exit(0));
    });
  })
  .catch((error) => {
    console.error("Error opening ngrok tunnel: ", error);
    process.exitCode = 1;
  });

Go back to package.json change the start script back to node ./bin/www and add a new script to run the application in dev mode:

  "scripts": {
    "start": "node ./bin/www",
    "dev": "node ./bin/dev"
  },

Now you can start your application with npm run dev and it will use nodemon to restart on file changes and open an ngrok tunnel which doesn't change when the application restarts.

Nodemon and ngrok working in tandem

You can adjust the script above to work for any of your applications. Indeed, aside from needing Node to run the nodemon and ngrok packages, you could use this for any application you are building. For more details, check out the nodemon documentation and the ngrok documentation.

If you are a VS Code user and you prefer having ngrok at the tip of your command prompt, take a look at my ngrok for VS Code plugin.

The script in this post was inspired by Santiago Palladino's version which I brought up to date and added usage instructions. Thanks to Santiago, Alex Bubenshchykov, the author of the ngrok Node package, Remy Sharp, the author of nodemon, and Alan Shreve, the creator of ngrok, all for making this possible.

How to display dates in your user's time zone with the Intl API

Mon, 22 Feb 2021 00:00:00 GMT

Time zones are hard. Not only are there a lot of them, but they don't fit nicely into whole hour blocks, daylight savings time changes individual zones some of the time, and zones move around and change all the time. In short, it is a hassle.

<figure> <img src="/posts/time_zones.png" alt="A map of time zones across the world" loading="lazy" /> <figcaption><a href="https://commons.wikimedia.org/wiki/File:World_Time_Zones_Map.png">Time zones of the world, via Wikimedia Commons</a>, aren't they nice and easy?</figcaption> </figure>

Events are happening online all the time and since people can join an online event from anywhere it is important to tell people what time that event will be happening for them. We need to combine an event's time, a user's time zone, and even their language and formatting preferences, to show a user a meaningful time.

Let's take a look at the Intl API, the ECMAScript Internationalisation API, and specifically Intl.DateTimeFormat which will aid us with date and time functions to achieve just that.

Get a user's time zone in JavaScript

There are two keys to getting a time correct in a user's time zone: the time in its original time zone and the user's time zone. If you have the time of an event, then it's up to you to provide that with its time zone, ideally in ISO 8601 format. But what can we do about getting the user's time zone?

Intl.DateTimeFormat is an object that can help us format the time. It also comes with a utility that tells us more about the user's time. Running Intl.DateTimeFormat().resolvedOptions() in a browser tells us everything the browser knows about the user's date and time preferences. If I run it for myself I get:

Intl.DateTimeFormat().resolvedOptions();
// => {
//   locale: "en-AU",
//   calendar: "gregory",
//   numberingSystem: "latn",
//   timeZone: "Australia/Melbourne",
//   year: "numeric",
//   month: "2-digit",
//   day: "2-digit",
// };

We can see that the browser knows I am in the "Australia/Melbourne" time zone. As I write this, that's Australian Eastern Daylight Time (AEDT), UTC+11. Between some time in April and October it will be Australian Eastern Standard Time (AEST), which is only UTC+10. See what I mean about this being hard? The place stays the same, but the time zone can change depending on the time of year. The good news is that Intl is aware of when that changes, so it can give us the time zone "Australia/Melbourne" and internally know when that translates to AEDT or AEST.

Now we have the time zone, we need to take a time and print it out for the time zone.

Showing a time in a time zone

The main job of Intl.DateTimeFormat is to format a time for a user. It takes a language and then a bunch of options, including the time zone, and returns a formatter function. That function takes a datetime object as an argument and returns a string that you can display to the user.

I can create a formatter that will format times for my time zone like this:

const formatter = Intl.DateTimeFormat("en-AU", { timeZone: "Australia/Melbourne" });

Then I can use that formatter to print out a date. I'll use the date I am writing this as an example, that way I can just use new Date().

const date = new Date();
formatter.format(date);
// => "22/02/2021"

You don't actually need to provide a language or time zone to the formatter, it will pick the system defaults.

const minimalFormatter = Intl.DateTimeFormat();
minimalFormatter.format(date);
// => "22/02/2021"

By default, the formatter outputs a short date form, even though the time is in there too. This is useful though, as we can still see that the time zone side of things is working. It may be the 22nd as I write this, but on the West coast of the USA it's still the 21st. We can see this by setting the time zone to something like "America/Los_Angeles":

const USFormatter = Intl.DateTimeFormat("en-AU", { timeZone: "America/Los_Angeles" });
USFormatter.format(date);
// => "21/02/2021"

A day behind, as expected. Note that Intl.DateTimeFormat takes a language string too. I've been using Australian English as the setting, but we can set this to US English too.

const reallyUSFormatter = Intl.DateTimeFormat("en-US", { timeZone: "America/Los_Angeles" });
reallyUSFormatter.format(date);
// => "02/21/2021"

Now we get the date in mm/dd/yyyy format, far and away the worst format. But, since Intl.DateTimeFormat knows international preferences for this, I can provide the language and the result will be formatted in the way the user expects and I never have to see a date with the month in the most significant place.

As an aside, there are a few ways to get a user's preferred language in the browser. navigator.languages returns an array of a user's preferred languages. navigator.language is supposed to return the first element of navigator.languages, but some browsers disagree and return the language of the browser UI, which is not necessarily the same. See more about the inconsistencies in this API on CanIUse.

Formatting the time

Now we know how to create a formatter that outputs a short date format, we need to know how to use it to format times the way we want. We saw that Intl.DateTimeFormat takes a language and then an object of options, including timeZone, to create a formatter. It is that object of options that we can use to add or remove parts of the date and time in the output. MDN has great documentation on the parameters for Intl.DateTimeFormat, but let's take a quick look at some options.

In the examples above, we gave no input for how we wanted to format the date. This is the equivalent of passing the defaults we saw above in the response to Intl.DateTimeFormat().resolvedOptions():

{
  day: "2-digit",
  month: "2-digit",
  year: "numeric"
}

The day, month and year options can be set to "numeric" or "2-digit" and month can also be "long", "short" or "narrow". Let's see what happens if we change things:

const newFormatter = Intl.DateTimeFormat("en-AU", {
  timeZone: "Australia/Melbourne",
  month: "long",
  year: "2-digit"
});
newFormatter.format(date);
// => "February 21"

In this case I defined month and year and left date off, so date no longer appears. Using "2-digit" for the year changes the output from "2021" to "21" and using "long" for the month prints out the full name of the month, "February".

We can add other elements to this format too, like weekday, era, hour, minute, second and timeZoneName. Here's a fully written out date time:

const fullFormatter = Intl.DateTimeFormat("en-AU", {
  timeZone: "Australia/Melbourne",
  day: "numeric",
  month: "long",
  year: "numeric",
  weekday: "long",
  era: "short",
  hour: "numeric",
  minute: "numeric",
  second: "numeric",
  timeZoneName: "short"
});
fullFormatter.format(date);
// => "Monday, 22 February 2021 AD, 5:05:52 pm AEDT"

There are a couple of shortcuts you can use too; timeStyle and dateStyle can be "full", "long", "medium" or "short" and you can use them together, but not with the above options.

const shortcutFormatter = Intl.DateTimeFormat("en-AU", {
  timeZone: "Australia/Melbourne",
  timeStyle: "long",
  dateStyle: "short"
});
shortcutFormatter.format(date);
// => "22/2/21, 5:05:52 pm AEDT"

When is your event?

Now, we can take all this knowledge and apply it to the date of an event. As I write this, it is the 22nd February, so let's consider an event in the future for me. Say it will occur on Friday 26th February, at 10am in the morning Pacific Time (UTC-8).

First we convert that time to ECMAScript compatible ISO 8601 format: 2021-02-26T18:00:00.000Z (Z means UTC with no offset, so 10am Pacific Time is 6pm in UTC). Other formats are supported by browsers, but by convention only, not as part of the standard.

We take the date string and create a new Date object with it. We also create a date formatter. We feed the date to the formatter and we get the date out in the correct time zone and format.

const excitingEventTime = "2021-02-26T18:00:00.000Z";
const eventDate = new Date(excitingEventTime);
const formatter = Intl.DateTimeFormat("en-AU", {
  timeZone: "Australia/Melbourne",
  day: "numeric",
  month: "long",
  year: "numeric",
  weekday: "long",
  hour: "numeric",
  minute: "numeric",
  second: "numeric",
  timeZoneName: "short"
});
formatter.format(eventDate);
// => Saturday, 27 February 2021, 5:00:00 am AEDT

So there we go, Friday morning events in Pacific Time happen early Saturday morning in AEDT.

Use Intl more

The Intl APIs, and in particular Intl.DateTimeFormat, make it easier for us as developers to display datetimes in the format, the time zone and the language of our users.

As a bonus and because Intl is part of ECMAScript, it is also available in Node.js. You will want to be more careful with defaults when working on server though. The time zone, for example, is going to be where the server is located, not the time zone of your user. I recommend working in UTC on servers and using users' time zones when displaying the datetime.

I recommend you familiarise yourself with everything that is available under Intl so you understand how you can use the platform to make your applications work better for your users wherever they are.

How to stream file downloads in Node.js with Got

Thu, 06 Aug 2020 00:00:00 GMT

Got is a Node.js library for making HTTP requests. It has both promise and stream based APIs and in this post I want to explore how to use the stream API to download files.

Using Got

If you use HTTP libraries for making API requests, then the promise method is likely the best for you. Making a basic HTTP request with Got looks like this:

const got = require("got");

got(url)
  .then(response => console.log(response.body))
  .catch(error => console.log(error.response.body));

The stream API gives us some extra benefits though. The promise API will load responses into memory until the response is finished before fulfilling the promise, but with the stream API you can act on chunks of the response as they arrive. This makes streams more memory efficient, particularly for large responses.

Streaming a file download with Got

You can create a stream with Got using the stream method or by setting isStream to true in the options.

got.stream(url);
// or
got(url, { isStream: true });

A Got stream is a duplex stream, which means it is both readable and writable. For the purposes of downloading a file, we will just be using its readable properties.

To download a file we need to send the response to the file system somehow. Streams allow you to pipe the data from one stream to another. To write to the file system we can create a writable stream using the fs module's createWriteStream.

To test this out we'll need a file we can stream. The URL in the following examples is a 500KB gif that you might like.

The simplest way to use a Got stream and write the file to the file system looks like this:

const got = require("got");
const { createWriteStream } = require("fs");

const url =
  "https://media0.giphy.com/media/4SS0kfzRqfBf2/giphy.gif";

got.stream(url).pipe(createWriteStream('image.gif'));

This code creates a Got stream of the image URL and pipes the data to a stream that writes the data into a file called "image.jpg".

Handling progress and errors

The above code will download the file as long as there are no problems. If an error occurs the code will crash with an unhandled exception. There is also no feedback, so if your file is large you will not see any result until the download is complete. We can listen to events on the stream to handle both of these cases.

Let's start by rearranging the code above. We'll get individual handles to the Got stream and the file writer stream.

const got = require("got");
const { createWriteStream } = require("fs");

const url = "https://media0.giphy.com/media/4SS0kfzRqfBf2/giphy.gif";
const fileName = "image.gif";

const downloadStream = got.stream(url);
const fileWriterStream = createWriteStream(fileName);

Now, before we pipe the downloadStream into the fileWriterStream attach some event listeners.

To get feedback on the progress of the download we can listen to the downloadProgress event on the downloadStream. The event fires with an object with 3 properties:

transferred: the number of bytes transferred so far
total: the total number of bytes
percent: the proportion of the transfer that is complete (between 0 and 1)

If the server you are downloading from doesn't return a Content-Length header for the file, then total will be undefined and percent will be 0 until the download is complete.

We can handle errors on both the downloadStream and fileWriterStream by listening for the error event. It is good to handle both of these errors as it gives us information over what failed. If the downloadStream emits an error then there is a problem with the URL, the network or the remote server. If the fileWriterStream emits an error then there is a problem with the file system.

For one last piece of feedback, we can also listen to the finish event on the fileWriterStream. This event is fired once all data has been written to the file system.

Let's complete the above code by adding these events and piping the downloadStream to the fileWriterStream.

const got = require("got");
const { createWriteStream } = require("fs");

const url = "https://media0.giphy.com/media/4SS0kfzRqfBf2/giphy.gif";
const fileName = "image.gif";

const downloadStream = got.stream(url);
const fileWriterStream = createWriteStream(fileName);

downloadStream
  .on("downloadProgress", ({ transferred, total, percent }) => {
    const percentage = Math.round(percent * 100);
    console.error(`progress: ${transferred}/${total} (${percentage}%)`);
  })
  .on("error", (error) => {
    console.error(`Download failed: ${error.message}`);
  });

fileWriterStream
  .on("error", (error) => {
    console.error(`Could not write file to system: ${error.message}`);
  })
  .on("finish", () => {
    console.log(`File downloaded to ${fileName}`);
  });

downloadStream.pipe(fileWriterStream);

If you run the above code in a terminal you will see your download progress logged to the terminal and the image will be downloaded successfully.

Getting fancy with more stream methods

Using streams to download files is more efficient than Got's promise methods, but the code above has taken a bit of a backward step in terms of developer experience. Rather than dealing with promises, which could be simplified with async/await, we now have to handle events with calbacks.

We can get back to this experience using the Stream module pipeline function. pipeline takes a number of streams as arguments and pipes the data between them. It also takes a callback function which is called if there is an error within the pipeline or once the pipeline is finished.

This still deals with callbacks, but we can use the Util module's promisify function to turn it into a promise.

Putting this together, we can simplify the above code to the following:

const got = require("got");
const { createWriteStream } = require("fs");
const stream = require("stream");
const { promisify } = require("util");
const pipeline = promisify(stream.pipeline);

const url = "https://media0.giphy.com/media/4SS0kfzRqfBf2/giphy.gif";
const fileName = "image.gif";

const downloadStream = got.stream(url);
const fileWriterStream = createWriteStream(fileName);

downloadStream.on("downloadProgress", ({ transferred, total, percent }) => {
  const percentage = Math.round(percent * 100);
  console.error(`progress: ${transferred}/${total} (${percentage}%)`);
});

pipeline(downloadStream, fileWriterStream)
  .then(() => console.log(`File downloaded to ${fileName}`))
  .catch((error) => console.error(`Something went wrong. ${error.message}`));

Or adding in async/await for the final step:

(async () => {
  try {
    await pipeline(downloadStream, fileWriterStream);
    console.log(`File downloaded to ${fileName}`);
  } catch (error) {
    console.error(`Something went wrong. ${error.message}`);
  }
})();

Node streams are cool 😎

Downloading a file is just one use of Node streams, you can find streams popping up all over the place. In this post we used a readable stream to download the file and a writable stream to write it to disk. You can also create readable streams of files and, if you are making POST requests with Got, you can stream the upload of data too. Objects like process.stdin, process.stdout and process.stderr are streams, as are HTTP requests and responses.

For more on streams, check the Node.js stream documentation and, for more in depth understanding, this guide on backpressuring in streams.

I built a VSCode extension: ngrok for VSCode

Tue, 14 Apr 2020 00:00:00 GMT

Over the Easter weekend, a four day weekend characterised by lockdowns all over the world, I decided to use the extra time I had at home to start a new project and learn a new skill. By the end of the weekend I was proud to release my first VSCode extension: ngrok for VSCode.

What's that now?

ngrok is a command line tool built by Alan Shreve that you can use to expose your localhost server with a publicly available URL. It's great for sharing access to an application running on your own machine, testing web applications on mobile devices or testing webhook integrations. For example, I'm a big fan of using ngrok to test my webhooks when I am working with Twilio applications.

VSCode is my current favourite text editor, built by Microsoft and based on JavaScript (well, mostly TypeScript).

As I was using VSCode last week I wondered if there was an extension that made it easier to use ngrok. I had a search and found one under development and one that started a web server as well as running ngrok. So I decided to build the extension I wanted to see in the marketplace.

What does it do?

With version 1 of the extension you can start an ngrok tunnel with either a port number or by choosing one of your named tunnels from your ngrok config file. There is one available setting, where you can set a custom path to a config file.

Once a tunnel is running you can then open the ngrok dashboard or close the tunnel.

All the commands are available from the VSCode command palette.

It's simple so far, but I wanted to keep the scope small and get it released.

The code is all open source and you can find it on GitHub.

What's next?

I would love for you to try the extension out, especially if you are already an ngrok user. If it's useful then I am looking for feedback, bug reports and feature requests so I can continue to improve it.

One idea I have already is to provide a Status Bar Item or Tree View that can give more information on and control over currently running ngrok tunnels. I should probably work out how to write tests for the extension too.

What do you think?

I really am after feedback, so please install ngrok for VSCode and let me know what you think on Twitter or via a review on the VSCode Marketplace.

Making a responsive Twitch Embed

Mon, 23 Mar 2020 00:00:00 GMT

I've been trying out streaming live code on Twitch which is a lot of fun. I wanted to share on my own site that I was streaming so I have built a page dedicated to it. The page will evolve, but one of the first things I wanted to include on it was an embed of my Twitch stream and chat.

Twitch makes it easy to embed your stream but the less obvious part is how to make that embed responsive across any browser size. So here's how to create a responsive Twitch embedded stream.

The HTML

There are 3 ways you can embed Twitch on your site, embedding everything, which involves loading the Twitch Embed JavaScript that creates an <iframe> on the page, embedding just chat and embedding just video, both of which use simple <iframe>s. Initially I thought I wanted to embed everything, but the video and chat within the <iframe> behaved weirdly and I didn't have any use for the interactive JavaScript library. Instead I plumped for embedding both the video and chat <iframe>s which turned out gave me much more control over the outcome.

The HTML that I used looks like this:

<div class="twitch">
  <div class="twitch-video">
    <iframe
      src="https://player.twitch.tv/?channel=phil_nash&parent=philna.sh&autoplay=false"
      frameborder="0"
      scrolling="no"
      allowfullscreen="true"
      height="100%"
      width="100%">
    </iframe>
  </div>
  <div class="twitch-chat">
    <iframe
      frameborder="0"
      scrolling="no"
      src="https://www.twitch.tv/embed/phil_nash/chat?parent=philna.sh"
      height="100%"
      width="100%">
    </iframe>
  </div>
</div>

The layout

The video is in 16:9 aspect ratio. On mobile I want the chat to sit below the video and then when the screen becomes wide enough on the right hand side of the video. When side-by-side, the chat and video should be the same height.

The tricky part here is the aspect ratio. CSS doesn't have a native way to accomplish aspect ratios yet. For now the way to force an aspect ratio in an element is unintuitive, but at least it works. You can apply this to any HTML you need to maintain a height that is relative to its width, things like <video> elements, or <iframe> embeds like YouTube or Twitch, as in this case.

Aspect ratios with CSS

The trick is to abuse how padding works on block elements. When you use percentage based top or bottom padding, that percentage is based on the width of the element. If we set the height of an element to 0, the width to 100% and the top padding to 75% then the element achieves an aspect ratio of 4:3. Change the padding to 56.25% and you have a 16:9 ratio.

I told you it was unintuitive. But it works.

You might have noticed that if we are just padding the top of the element then there is no actual space for the content. We solve this with absolute positioning, placing the <iframe> back at the top of the element and giving it a height and width of 100% to fill out the space.

With the above HTML, the following CSS lays out the <iframe> with a 16:9 aspect ratio:

.twitch .twitch-video {
  padding-top: 56.25%;
  position: relative;
  height: 0;
}

.twitch .twitch-video iframe {
  position: absolute;
  width: 100%;
  height: 100%;
  top: 0;
}

To complete the basic layout the chat <iframe> appears below the video and behaves as a regular block level element. I set a height to give the chat enough vertical space:

.twitch .twitch-chat {
  height: 400px;
}

.twitch .twitch-chat iframe {
  width: 100%;
  height: 100%;
}

To create the side-by-side layout at wide screen sizes we need to alter this a bit. The video will take up 75% of the width of the container and the chat 25%. This is adjusted within a media query to only take effect above a certain width, for my site this became 850px.

When making the width of the video container 75%, you have to adjust the padding with the same ratio. To maintain the 16:9 aspect ratio the top padding becomes 42.1875%.

To keep the chat window the same height as the video, a little more absolute positioning is put into effect. The wrapper of both children is positioned relatively, then the chat <div> is set to 25% width and placed absolutely on the right side of the wrapper:

@media screen and (min-width: 850px) {
  .twitch {
    position: relative;
  }

  .twitch .twitch-video {
    width: 75%;
    padding-top: 42.1875%;
  }

  .twitch .twitch-chat {
    width: 25%;
    height: auto;
    position: absolute;
    top: 0;
    right: 0;
    bottom: 0;
  }
}

The result

With the above HTML and CSS, the output ends up looking like this:

Correct aspect ratios and responsive layout; mission accomplished. Check it out live here.

If you want to use this on your own site, feel free to take this HTML and CSS. Just remember to replace the channel name in the embed src. You can also learn more about the CSS aspect ratio trick and other ways to accomplish the effect at CSS Tricks.

And of course, please join me on Twitch where I will be chatting development, live coding a variety of projects, and generally having fun.

The story of a mildly popular Ruby gem

Tue, 17 Mar 2020 00:00:00 GMT

The list on GitHub of repositories that depend on your repository is scary.

There's something nice about writing and releasing a library, module, Ruby gem, whatever. It is code that you wrote that worked for you. There may be download numbers on the registry, but they are fairly abstract and don't necessarily relate to use. Seeing that there are real projects using the code that you wrote is kind of terrifying. More so when some of that code is still around from 10 years ago.

The first days of the Bitly Ruby gem

I started out in the web development industry as a front-end developer. I joined a company that built sites in Ruby on Rails and eventually worked my way from working with views to controllers, then models, background jobs, mailers. The whole application stack. At that point I believed myself to be a Rails developer. I had never written any plain Ruby though. In an effort to change this, I decided I was going to write and release a gem.

As luck would have it I liked working with APIs, was building a lot of social applications at work at the time and the era of Twitter's serious 140 characters and link shortening was upon us. Bitly announced they were releasing a new API and I took this as a chance to write my first Ruby. I made my first commit to the GitHub repo on 21st January 2009.

It appears that the first commit included enough code to shorten a URL with the API. The tests were not good. Five days later it had grown into something I deemed worth releasing. On the 26th January 2009 I released version 0.1.0 into the world.

On the 27th January 2009 I released version 0.1.1 because I'd forgotten a dependency and version 0.1.0 just didn't work.

Notably these releases came before Rubygems.org existed. (Remember those days? Me neither.) You cannot find these early releases available there, but that's probably for the best!

It takes on a quiet life of its own

The beauty of open source software is that when you put something out into the world and other people find it useful they will also contribute back. Over the course of 2009 I received contributions from 7 different developers improving and adding functionality to this code. In 2010 I released version 0.5.0 which had support for Bitly's version 3 API (no, I was not very good at semantic versioning back then).

For the next 8 years, the gem didn't need a lot more work. I guess it just worked for its users. There were some feature additions and bug fixes from those that used it. In total, over the life of the project so far, there have been 17 contributors other than me. The project has never been perfect, it was my first after all, but it did the job for plenty of people.

Your code is not you

...but it's hard to remember that at times. The Bitly project has long been my most starred project on GitHub, it sits at the top of my pinned repositories on my profile page and has the highest number of downloads on rubygems.org of my own projects. I'm proud of it, but up until recently there was still code in that project from 2009. It always niggled at the back of my mind that, even though the gem worked, it wasn't a good representation of my ability to code.

For this reason I embarked on a rewrite of the gem from the ground up in 2018. It was hard to motivate myself to complete it though, in fact I didn't get much beyond the OAuth2 implementation. It turns out that replacing working code is not a good reason to rewrite things, even if you don't like the existing code. In my head I was torn between rebuilding this project with the experience of 10 more years writing Ruby against using my time for other things that were important to me. Mostly my time won out, but it didn't stop me thinking about it.

Then in 2019 Bitly introduced the 4th version of the API. While that meant the rewrite pivoted from re-implementing the version 3 API, it still didn't drive motivation. Further, Bitly announced they would be stopping support for the version 3 API on the 1st March 2020.

Eventually that got me going and I managed to put the work in to put together what has just been released as version 2 of the Bitly Ruby gem. I'm happier with the library I have put together this time, it is better written, better tested and better documented than ever before. It is once more a project I can be proud of and I hope that many more projects can happily integrate it for their link shortening requirements.

The open source iceberg

There are many differences between a Ruby project like the Bitly gem and Rails, Devise, or even Jekyll. So many open source projects are driven by a single maintainer and rely on their time and attention. They aren't core to applications, but they make developers' lives easier. There's a lot of talk in the open source world about how to sustain the large projects that I think ignores the mildly popular projects. Much of that talk about sustainability focuses on making open source easier for under-represented developers to get into, open source shouldn't be the bastion of those with enough time and money to dedicate some of it to work on projects like this. But there are considerably more small projects than the flagship libraries and frameworks that can attract funding and those projects will continue to be built by those with expendable time and effort. I don't have any answers here, I just wanted to draw a little attention to the many, many open source projects in this position.

I have more code to write

I feel very fortunate to be able to find time to work on projects that allow me to practice my craft and potentially make another developer's life easier. The Bitly gem was my first foray into the world of creating an open source project and it lives on today. Bitly have removed their support for the version 3 API so there should be plenty of developers looking to upgrade their implementations or the libraries they depend on. Hopefully version 2 will suit them, maybe there will be more work in this gem for me yet.

The list on GitHub of repositories that depend on your repository is scary.

Mistakes I've made treating file paths as strings

Wed, 04 Mar 2020 00:00:00 GMT

Some things you do as a developer can work for you for years, then turn around and bite you when you were least expecting. These are the things that you wish another developer had told you early in your career so you never had to make the mistakes. This post is about one of those things and if you're reading this, consider it me telling you.

File paths look like strings. You have a number of directories and maybe a file name with an extension at the end. You separate the directories and files with a / character and the result looks like /path/to/file. So you can treat them like strings, joining them or concatenating them until you pass them to another file method that is used to read from or write to the file. These were my thoughts from just a few months ago. Here's where I was wrong.

Don't forget Windows

If you develop on a Mac, like I have the privilege of doing, or Linux then you might have read the above paragraph and not noticed anything wrong. If develop on Windows you probably sighed into your cup of coffee as you read the / character.

It's all too easy to forget when you work with a Mac and deploy to Linux environments, like I have done for years, that Windows uses backslashes. It's all too painful to find out you've made that mistake when you work on a command line tool that needs to run on both types of platform. create-twilio-function is one such command line tool that had to go through a number of changes from things like:

mkdir(path + '/' + dirName);

const path = require('path');
mkdir(path.join(pathName, dirName));

so that it would work properly on Windows.

To Windows users, I'm sorry. To everyone else, when working with Node.js the path module is your friend. Use path.join whenever you have to join two paths. And check out other utilities like path.relative, which returns a relative path from one path to another, and path.normalize, which returns a path resolving segments like . or ...

Pay no attention to path.sep, which returns a / or a \ depending on the system you're working on, just use path.join.

Paths behave differently to strings

To my second mistake, this time working in Ruby. This one was slightly more subtle and evaded my tests. You see, you can use the Pathname class to create fragments of paths and then concatenate them. For example:

require "pathname"
path1 = Pathname.new("path")
path2 = Pathname.new("to")
path1 + path2
# => #<Pathname:path/to>

As you can see Pathname objects have a + operator that concatenates the paths, much like + concatenates strings. In fact, it also works with a mix of strings and paths:

require "pathname"
path1 = Pathname.new("path")
path2 = "to"
path1 + path2
# => #<Pathname:path/to>

This all seems well and good, except it doesn't work the other way around.

require "pathname"
path1 = "to"
path2 = Pathname.new("path")
path1 + path2
# => TypeError (no implicit conversion of Pathname into String)

A nice error like that means we've done something wrong, that was not the problem I had though. No, the issue I had stemmed from expecting to concatenate a pathname and a string and instead concatenating two strings. This manifested itself in my Rubygem jekyll-gzip. You see, I was trying to create a glob of paths with the line:

files = Dir.glob(dir + "**/*{#{extensions}}")

It turned out under some circumstances dir was actually a string instead of a pathname and it didn't include a separator. So the glob was looking for "dirname**/*{#{extensions}}" when I really wanted it to look for "dirname/**/*{#{extensions}}". Concatenating two pathnames or a pathname and a string will add the separator (as someone pointed out in a comment on my commit), but concatenating two strings will not. This meant that the gem happily went looking for the wrong pathname, found no files and then proceeded to successfully do nothing. Replacing the entire line with:

files = Dir.glob(File.join(dir, "**", "*{#{extensions}}"))

fixed the issue. In this case File.join is the method to use to avoid surprises with strings.

Always use the built in path tools

Whether you're working in Node.js, Ruby, or any other language do not be tempted to treat file paths as strings. They behave differently on different platforms and mixing paths and strings together can cause hard to debug errors.

Use your standard library and save yourself the hassle.

How to find CFPs for developer conferences

Wed, 29 Jan 2020 00:00:00 GMT

So you've decided to speak at a developer conference? You have a story you want to share with your peers—how you built something, how you learned something new, how you became a better developer and how everyone else can too—but you need to find a stage on which to share this story.

There are hundreds of developer conferences out there, covering everything you could think of in our industry. Knowing they are there is nice, but finding out which are coming up and are have an open call for proposals (CFP) is the challenge. Over a number of years of submitting to many and getting accepted to a few conferences, I have researched a number of resources for finding CFPs and I want to share the current crop in this post.

Websites, mailing lists, Twitter accounts

Ideally all CFPs that were relevant to your interests or communities would be sent directly to you. Sadly that's not how the world works. Instead, we are fortunate that a number of people and communities do the hard work to go out and find the available CFPs and aggregate them for us. The following list includes websites, mailing lists and Twitter accounts that share developer conferences and their CFPs. Mailing lists bring the CFPs to your inbox, Twitter accounts are great if you're all over social media and the sites are always worth browsing.

Read on to discover how to find developer conference CFPs.

Probably the most comprehensive website and mailing list full of developer CFPs that I know today, CFP Land has a great list of conference CFPs that you can filter by category, region, date and whether they offer hotel, travel or a stipend for speakers. CFP Land also sends out a round up of CFPs weekly and you can sign up for the mailing list on the CFP Land home page.

As a bonus it also has a Twitter account, RSS feed and even a CFP API.

One thing to consider is that CFP Land offers a professional account. The basic website and the other services listed above list CFPs that are closing within 21 days. The professional account gives you earlier access to the CFPs seen on the public site as well as tools to save CFPs you're interested in and reminders for when the deadlines are approaching. While I am not currently a customer and I can't comment on how much more useful this is, it is something I have considered and mostly I appreciate the effort to make CFP Land sustainable for the future.

Tech Daily CFP is a Twitter account that retweets developer CFPs and a mailing list that gathers those retweets and mails a collection of them once a day.

It can be quite intense to receive emails once a day with CFPs, but it definitely feels like you're not missing out on anything.

PaperCall is a site on which you can host your conference's CFP, so naturally it has a list of currently open CFPs that is always worth perusing. PaperCall used to run the very comprehensive WeeklyCFP mailing list, but I haven't heard from it for a while so I believe it's defunct (which is certainly a reason to support CFP Land's professional accounts).

Confs.tech is a community sourced list of conferences. They also list live CFPs that you can filter by category and country. They tweet out all the new conferences added to the site and I just discovered you can sign up to their mailing list on their home page

I came across SeeCFP on the day of writing this post, so I'm not very well acquainted with it yet. It has a list of CFPs stored in Airtable and a weekly mailing list you can sign up to on the home page. They also tweet out CFPs that are close to their deadlines.

Colloq is more than just a list of CFPs, it's a social event platform. It lists loads of events, including events with open CFPs but also lets you publicly say you are attending, links to ticket sales and collects coverage of the event after it happens (check out Accessibility Club Summit 2019 for a good example of an event on Colloq).

Colloq is also a good example of an application that has been built with privacy and security at its heart (if only I could say that about more applications). Just read their blog post on how they keep user passwords secure to see the level of attention.

Tulula lists events, particularly those with open CFPs and has a comprehensive filtering system to find the events that are important to you. It also includes a forum and Slack community so that you can meet and chat with others trying to improve their public speaking too.

Other lists of conferences

There are a bunch more sites or Twitter accounts that are less comprehensive or that I don't necessarily use but came across as part of my research. Some are specific to languages or regions and any of them might be useful to browse once in a while.

Good luck!

Finding the right conference for your talk is a challenge and hopefully these resources can help you to discover a few more opportunities.

If I've missed a CFP resource, please let me know on Twitter.

Next up, watch out for my new mailing list sharing sites that list CFPs 😉.

Testing signed and encrypted cookies in Rails

Wed, 15 Jan 2020 00:00:00 GMT

Recently I've been refactoring the tests for a gem I maintain and I needed to test that it sets the right cookies at the right time. But the cookies in use in the gem are signed cookies and that caused a slight hiccup for me. I'd never tested the value in a signed cookie before and it wasn't immediately obvious what to do.

So I thought I would share what I found out in case it helps.

Cookies on Rails

In Rails applications there are three flavours of cookies available: simple session cookies, signed cookies and encrypted cookies. You can set any of these by using the cookies object in a controller, like this:

class CookiesController <  ApplicationController
  def index
    cookies["simple"] = "Hello, I am easy to read."
    cookies.signed["protected"] = "Hello, I can be read, but I can't be tampered with."
    cookies.encrypted["private"] = "Hello, I can't be read or tampered with."
  end
end

Simple cookies

Simple cookies are made up of plain text. If you inspected the cookie above called "simple" in the browser you would see the text "Hello, I am easy to read."

Simple cookies are ok for storing data that doesn't really matter. The end user can read and change it and your application shouldn't be affected.

Signed cookies

Signed cookies are not sent to the browser as plain text. Instead they comprise of a payload and signature separated by two dashes --. Before the dashes, the payload is base 64 encoded data. To read the data you can base 64 decode it. This data isn't secret, but it can't be tampered with because the second part of the cookie is a signature. The signature is created by taking an HMAC SHA1 digest of the application's secret_key_base and the data in the cookie. If the contents of the cookie are changed when you try to read the cookie, the signature will no longer match the contents and Rails will return nil. Under the hood this is all handled by the ActiveSupport::MessageVerifier. As you can see above, you don't need to worry about that, you can treat the cookies.signed object as if it were a hash.

Signed cookies are useful for data that can be read by the user, but you need to trust is the same when you get it back to the server again.

Encrypted cookies

Encrypted cookies take this one step further and encrypt the data in the cookie, then sign it. This is handled by the ActiveSupport::MessageEncryptor and means that without the secret_key_base you cannot read or write to this cookie. Thankfully there's no need to worry about the encryption yourself, using the cookies.encrypted object you can set encrypted cookies as though they were a regular hash.

Encrypted cookies are useful for private data that you want to store with the user, but you don't want them, or anyone, to read.

Testing cookies

Suppose we now want to test the controller we saw above. We want to ensure that all of our cookies are set correctly. The test might look something like this:

class CookiesControllerTest < ActionDispatch::IntegrationTest
  test "should set cookies when getting the index" do
    get root_url
    assert_response :success
    assert_equal "Hello, I am easy to read.", cookies["simple"]
    assert_equal "Hello, I can be read, but I can't be tampered with.", cookies["protected"]
    assert_equal "Hello, I can't be read or tampered with.", cookies["private"]
  end
end

Or with RSpec Rails:

RSpec.describe CookiesController, type: :request do
  it "should set cookies when getting the index" do
    get root_url
    expect(response).to have_http_status(:success)
    expect(cookies["simple"]).to eq("Hello, I am easy to read.")
    expect(cookies["protected"]).to eq("Hello, I can be read, but I can't be tampered with.")
    expect(cookies["private"]).to eq("Hello, I can't be read or tampered with.")
  end
end

But this would fail at the test for the signed cookie and wouldn't pass for the encrypted cookie either. You can't just call on those cookies straight out of the jar if they have been signed or encrypted.

You might think you should test against the signed and encrypted version of the cookies, like this:

    assert_equal "Hello, I can be read, but I can't be tampered with.", cookies.signed["protected"]
    assert_equal "Hello, I can't be read or tampered with.", cookies.encrypted["private"]

That doesn't work either. At least it doesn't work if you are using the currently recommended way of testing controllers, with ActionDispatch::IntegrationTest in Minitest or type: :request in RSpec.

If you have the older style ActionController::TestCase or type: :controller tests, then cookies.signed and cookies.encrypted will work. If you have an application with the older style tests, do carry on reading just in case you decide to refactor them to come in line with the current Rails way.

With the tests above, the cookies object is actually an instance of Rack::Test::CookieJar, which does not have knowledge of your Rails application secrets.

So how do we test these cookies?

This is where I got to with the gem I was working on. I needed to test the result of a signed cookie, but I had a Rack::Test::CookieJar object. The good news is we can bring the Rails application's own ActionDispatch::Cookies::CookieJar back into play to decode your signed cookies and decrypt your encrypted cookies.

To do so, you instantiate an instance of ActionDispatch::Cookies::CookieJar using the request object from the test and a hash of your cookie data. You can then call signed or encrypted on that cookie jar. So now the test looks like:

class CookiesControllerTest < ActionDispatch::IntegrationTest
  test "should set cookies when getting the index" do
    get root_url
    assert_response :success
    assert_equal "Hello, I am easy to read.", cookies["simple"]
    jar = ActionDispatch::Cookies::CookieJar.build(request, cookies.to_hash)
    assert_equal "Hello, I can be read, but I can't be tampered with.", jar.signed["protected"]
    assert_equal "Hello, I can't be read or tampered with.", jar.encrypted["private"]
  end
end

Or the spec would look like:

RSpec.describe CookiesController, type: :request do
  it "gets cookies from the response" do
    get root_url
    expect(response).to have_http_status(:success)
    expect(cookies["simple"]).to eq("Hello, I am easy to read.")
    jar = ActionDispatch::Cookies::CookieJar.build(request, cookies.to_hash)
    expect(jar.signed["protected"]).to eq("Hello, I can be read, but I can't be tampered with.")
    expect(jar.encrypted["private"]).to eq("Hello, I can't be read or tampered with.")
  end
end

Red, Green, Re-snack-tor

In this post we've seen how to test signed or encrypted cookies in Rails. Hopefully your test suite is running green and your cookies are covered now.

I'm going to get back to the refactor I was working on. There are plenty more tests to cover now that these cookies have been polished off.

How not to sort an array in JavaScript

Mon, 26 Aug 2019 00:00:00 GMT

Array sorting is one of those things you don't spend too long thinking about, until it stops working for you. Recently I was working with array of items in JavaScript that were not sorting at all properly and completely messing up an interface. It took me way too long to work out what went wrong so I wanted to share what happened and why it was so weird.

Basic sorting

JavaScript has a sort method available on Array objects and running it will probably do what you expect. For example:

const stringArray = ['cat', 'dog', 'ant', 'butterfly'];
stringArray.sort();
// => [ 'ant', 'butterfly', 'cat', 'dog' ]

It's even pretty good if you're sorting arrays that might have members that are undefined. MDN says that "all undefined elements are sorted to the end of the array.".

const stringArrayWithUndefined = [
  'cat',
  undefined,
  'dog',
  undefined,
  'ant',
  'butterfly',
  'zebra'
];
stringArrayWithUndefined.sort();
// => [ 'ant', 'butterfly', 'cat', 'dog', 'zebra', undefined, undefined ]

Gotchas

The first issue you might come across is if you find yourself with an array containing null.

const stringArrayWithUndefinedAndNull = [
  'cat',
  undefined,
  'dog',
  undefined,
  'ant',
  null,
  'butterfly',
  'zebra'
];
stringArrayWithUndefinedAndNull.sort();
// => [ 'ant', 'butterfly', 'cat', 'dog', null, 'zebra', undefined, undefined ]

Sorting will coerce the null to the string "null" which will appear somewhere in the middle of the alphabet.

Then there are numbers. The default JavaScript sorting algorithm is to convert all members of an array to strings and then compare their sequences of UTF-16 code unit values. This works great for arrays of strings as we've already seen, but it breaks down very quickly for numbers.

const numberArray = [5, 3, 7, 1];
numberArray.sort();
// => [ 1, 3, 5, 7 ]

const biggerNumberArray = [5, 3, 10, 7, 1];
biggerNumberArray.sort();
// => [ 1, 10, 3, 5, 7 ]

In the example above, 10 gets sorted to before 3 because "10" is sorted before "3".

We can fix this by providing JavaScript a comparison function to use to perform the sort. The function receives two items from the array and it needs to return a numeric value and whether that value is above, below or equal to zero defines how the items are sorted relative to each other. If the return value is less than zero, then the first item is sorted in front of the second, if the value is above zero then the second item is sorted in front of the first. If the return value is 0 then the items stay in the same order with respect to each other.

To sort numbers in ascending order, the comparison function is relatively simple:

const compareNumbers = (a, b) => a - b;

Subtracting the first item from the second one satisfies the requirements above. Using this comparison function with our biggerNumberArray from earlier will sort the numbers correctly.

biggerNumberArray.sort(compareNumbers);
// => [ 1, 3, 5, 7, 10 ]

This still works if you have undefined elements as they are ignored and sorted to the end.

const numberArrayWithUndefined = [5, undefined, 3, 10, 7, 1];
numberArrayWithUndefined.sort(compareNumbers);
// => [ 1, 3, 5, 7, 10, undefined ]

null causes problems again though.

const numberArrayWithUndefinedAndNull = [5, undefined, 3, null, 10, 7, 1];
numberArrayWithUndefinedAndNull.sort(compareNumbers);
// => [ null, 1, 3, 5, 7, 10, undefined ]

This happens because coercing null to a number returns 0.

Number(null);
// => 0

You could handle this in your compareNumbers function or be happy that it is consistent.

Inconsistent gotchas

The biggest problem, and this caught me out recently, comes when undefined sneaks in another way. As we've seen, if the array contains undefined it's ignored and just sorted to the back. However, if you are sorting objects where the keys may be undefined this automatic sorting doesn't happen and the results become inconsistent.

For example, if you have an array of objects where some of them have values and some don't, trying to sort by that value will not give you the result you want.

const objectArray = [
  { value: 1 },
  { value: 10 },
  {},
  { value: 5 },
  { value: 7 },
  { value: 3 }
];
const compareObjects = (a, b) => a.value - b.value;
objectArray.sort(compareObjects);
// => [ { value: 1 },
//      { value: 10 },
//      {},
//      { value: 3 },
//      { value: 5 },
//      { value: 7 } ]

Subtracting a number from undefined or subtracting undefined from a number both return NaN and since that doesn't lay on the scale of numbers that sort needs from the comparison function the results end up a little strange. In this case, the item that caused the problem stays where it started in the array and the other objects are locally sorted.

There are a few ways around this, but the important thing is knowing that it can happen. In my case when I came across this, I filtered out the items that didn't have a value as they weren't important until they did.

objectArray
  .filter(obj => typeof obj.value !== 'undefined')
  .sort(compareObjects);
// => [ { value: 1 },
//      { value: 3 },
//      { value: 5 },
//      { value: 7 },
//      { value: 10 } ]

Beware sorting

The upshot of all of this is that the sort function is not as straightforward as it might seem. Strings work, numbers need some input and while undefined is handled as a primitive you have to keep an eye on coercing nulls or undefined object values.

Have you come across problems sorting in JavaScript or other languages? I'd love to hear your stories too, so give me a shout on Twitter at @philnash.

How to start a Node.js project

Thu, 10 Jan 2019 00:00:00 GMT

Sometimes I write blog posts to remind myself what I've learned and sometimes I write them because someone else shares something and I want to remember that better. This post is one of the latter.

Starting a Node.js project

Usually when I start a new Node.js project I use npm to generate my initial project.

npm init

npm then asks me some questions and builds a package.json file for me. Then I start building the project.

Later I inevitably copy and paste a .gitignore file from GitHub's useful repo of .gitignore templates. And if I remember I'll actually create a LICENSE file with the open source license that I intended to use.

This is not efficient.

Then this week I saw Tierney Cyren tweet this:

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">How to start any new Node.js project:<br><br>$ npx license mit > LICENSE<br>$ npx gitignore node<br>$ npx covgen YOUR_EMAIL_ADDRESS<br>$ npm init -y<br><br>You're ready to start coding.</p>— Tierney Cyren (@bitandbang) <a href="https://twitter.com/bitandbang/status/1082331715471925250?ref_src=twsrc%5Etfw">January 7, 2019</a></blockquote>

These four commands do everything that I was doing manually and more, setting up a project for success right from the start.

npx license mit uses the license package to download a license of choice, in this case the MIT license
npx gitignore node uses the gitignore package to automatically download the relevant .gitignore file from GitHub's repo
npx covgen uses the covgen package to generate the Contributor Covenant and give your project a code of conduct that will be welcoming to all contributors

If you've not seen npx before it looks locally to see if there is a command to run and executes it, if there is no local command it will try to download, install the command from npm, and run it. This is really useful when generating new projects and saves you from globally installing a bunch of npm packages that are only used in this setup mode.

npm init -y accepts all of the default options that npm init asks you about

Tierney also suggested customising your npm init defaults so that the output of npm init -y is correct.

Customising `npm init`

You can see your current npm config by entering npm config list on the command line. To just see the config that affects npm init you can grep for "init":

npm config list | grep init

There are a number of defaults you can set; author name, author email, author url, the license, and the version. To set them, you can enter them on the command line or use npm config edit to open up the config file in your text editor. The command line is easy enough though, you can set all five defaults like so:

npm set init.author.name "Your name"
npm set init.author.email "your@email.com"
npm set init.author.url "https://your-url.com"
npm set init.license "MIT"
npm set init.version "1.0.0"

Once you have that customised to your liking, npm init -y will always produce the right settings.

Building your own init script

There are some improvements that I'd make to Tierney's commands, though I appreciate they were constrained by Twitter. Here's a bash script I have come up with inspired by their tweet.

function node-project {
  git init
  npx license $(npm get init.license) -o "$(npm get init.author.name)" > LICENSE
  npx gitignore node
  npx covgen "$(npm get init.author.email)"
  npm init -y
  git add -A
  git commit -m "Initial commit"
}

To the original I've added fetching the license type, the author name and email from the npm init defaults. I've also initialised a new git repository and committed the results of this script as the "Initial commit".

You can take this function and add it to your ~/.bash_profile. Then, either source ~/.bash_profile or open a new command line window and run node-project. Feel free to add or remove other bits as you see fit to create your perfect initialisation script.

Go start a project

Now you have the perfect script to start a Node.js project why not go create a new one? I have a few small projects in mind that I plan to build this year and this is a nice basis to start from.

If you have any more suggestions to improve the script, let me know on Twitter at @philnash. Happy open sourcing!

2018 in review

Thu, 03 Jan 2019 00:00:00 GMT

I've never been one for writing a review of my year. This year I've found myself not only reading, but comparing myself to others' reviews. I realised that this is not particularly helpful, so decided I'd give myself something to compare myself to at the end of 2019. Thus, here is my first year in review.

Here are some of the things I was proud of this year:

Professional

July 2018 saw me cross 4 years of working as a developer evangelist for Twilio. It's been a really fun, interesting, hard, tiring and compelling role over this time. I'm almost half way through my 5th year now and I'm having a great time sharing my development experiences, helping other developers and being a part of the Twilio community.

Development

GitHub tells me I made 582 contributions in 2018. Those contributions were a good mix of personal projects, example Twilio applications and contributions to other open source projects. They also resulted in the release of 5 libraries; 4 in Ruby and one in JavaScript. They were:

pwned: a library to access the Pwned Passwords API with Ruby
Three Ruby libraries for compressing text files in Jekyll sites:
web-share-wrapper: a web component to replace share links with the web share API

I also began work to resurrect my RubyGem for the Bitly API as it has fallen way out of date. I need to spend more time with this though, the going has been slow so far.

At work I took over maintenance of the Authy-Devise library. The library makes it easy to implement two factor authentication in Devise with Twilio's Authy API. There's some good opportunities to improve the library this year, especially in testing. There are some interesting issues with testing a library that plugs in to a library that relies on Rails.

Helping

I like to help out on Stack Overflow. At the end of 2018 my reputation was 37,030 which I am incredibly proud of. In 2018 I left 345 answers, of which 169 were accepted. I also left 918 comments on questions/answers either asking for more information or following up.

I am fortunate enough to do this as part of my work, as it can take up quite a bit of time. Most of my answers are for questions tagged "Twilio" though you'd be surprised by the variety of problems developers have within the bounds of including Twilio in an application.

Learning

In "exceptionally late to the party" news I managed to take some time in 2018 to start learning both React and Angular. I've never been a big fan of applications that just work in the front-end (unless absolutely necessary, you can't host a WebRTC video chat without JavaScript!) and frameworks have tended to encourage that. This has changed over time and frankly both React and Angular are too big to ignore any more (and arguably were well before 2018 too).

I've enjoyed kicking up my learning and there's going to be more of this, likely around React, for 2019. Look out for more blog posts as I discover new things.

Writing

This leads me to my writing this year. Before 2018 I wrote the last post on my blog in July 2017. I intended to be more consistent in my writing in 2018 and that sort of worked for the first 3/4 of the year. My last posts on both my blog and the Twilio blog were both in October. Still, the results aren't all bad. I wrote:

9 posts on philna.sh/blog
11 posts for Twilio
1 guest post for Rollbar

21 posts in total is good, but I want to improve on that this year. More consistency is the main aim again. Thankfully I have a number of drafts and projects already lined up.

Speaking

I spoke 12 times in 2018. This is a marked step down from 2017 and even further from 2016. This coincides with my move to Australia where there are fewer events and fewer nearby countries to pop over to give talks, something I was fortunate enough to do in the past.

That said, speaking is more about quality than quantity. I was lucky to be selected to speak at these wonderful conferences:

JSConf EU in Berlin
Web Directions Code in Melbourne
NDC Sydney
Voxxed Days Melbourne
Angular Conf Australia (learning Angular this year paid off!)
Melbourne DevFest

JSConf EU was certainly my highlight here, I've been trying for years to get a talk accepted and I was elated when I received the confirmation email. The talk was one of the more important ones I've given too, addressing developers' use of push notifications and permissions on the web. It might also be the best talk title I've ever come up with. You can see "Aggressive Web Apps" on YouTube.

I had a lot of fun also presenting at local meetups. 2018 included talks at MelbJS, The Web Meetup, RoRo Melbourne and RoRo Sydney.

Travel

In 2018 I visited Sydney, Brisbane, Hobart, Singapore, Phuket, Bali, London, Berlin, New York, San Diego and San Francisco. 11 different places and 6 countries (aside from Australia).

I certainly racked up some miles there. I enjoy travelling, it's both part of the job and how I like to holiday. It can be tiring, particularly when it is for work, but with enough recovery time built in this much travel is sustainable. Getting to travel back to Europe for JSConf EU and use that as an opportunity to call in on my friends and family in the UK was fantastic too.

I am looking forward to exploring more new parts of the world in 2019.

Personal

2018 was my first full year living in Australia. It's been a fantastic year, Melbourne is an amazing city full of great food, interesting beer, lot's to do and above all welcoming and friendly people. Kelly and I liked it so much we bought a house here this year! This, combined with getting my permanent residency, means that we're going to be enjoying life in Melbourne for a while longer.

In numbers

To summarise the above, here's what it looks like in numbers:

🏠 1 new house<br> 🛂 1 permanent residency<br> ☎️ 4 years at Twilio<br> 📝 21 blog posts<br> 🎙 12 talks given<br> 💻 582 GitHub contributions<br> 💎 5 new libraries released<br> 🏙 11 cities visited<br> 🌏 6 countries visited<br> 🍻 296 new beers

I hope you had a great 2018 and wish you all the best for 2019.

Service workers: beware Safari's range request

Tue, 23 Oct 2018 00:00:00 GMT

You've implemented a service worker to cache some assets. Everything is working well, your service worker is a success, you're feeling good. But then...

Some time passes and you deploy a video to your site. Everything is still working well in Chrome, in Firefox, in Edge. You check Safari. The video is broken. You don't know what's gone wrong.

That was me last month. I published an article on animating in the canvas with React and wanted to use an animation as the header image. A video was much smaller than using a comparable gif so I implemented the header as a video. When I came to check things in Safari I found it had stopped working and I had no idea why.

Diagnosing the problem

I first thought it could have something to do with the CDN I'm using. There were some false positives regarding streaming video through a CDN that resulted in some extra research that was ultimately fruitless. Once I'd exhausted that line of inquiry I went back to the failing request.

Observing the request in Safari's inspector lead to further trawling the internet and eventually things started to add up. Safari was sending an initial request to fetch the video with a Range header set to bytes=0-1. You see, Safari requires HTTP servers that are serving video and audio to support Range requests like this.

Nginx was serving correct responses to Range requests. So was the CDN. The only other problem? The service worker. And this broke the video in Safari.

Time to fix the service worker then. With inspiration from this example of caching videos from the Chrome team, here's what I did.

Range requests in Service Workers

My existing service worker implementation was checking for requests to my assets or images directories and responding with a cache then network strategy.

self.addEventListener('fetch', function(event) {
  var url = new URL(event.request.url);
  if (url.pathname.match(/^\/((assets|images)\/|manifest.json$)/)) {
    event.respondWith(returnFromCacheOrFetch(event.request, staticCacheName));
  }
  // other strategies
});

The video requests are going to be served from the assets or image directories, so this is the place to interject. I checked for the existence of the range header and responded with a different approach.

self.addEventListener('fetch', function(event) {
  var url = new URL(event.request.url);
  if (url.pathname.match(/^\/((assets|images)\/|manifest.json$)/)) {
    if (event.request.headers.get('range')) {
      event.respondWith(returnRangeRequest(event.request, staticCacheName));
    } else {
      event.respondWith(returnFromCacheOrFetch(event.request, staticCacheName));
    }
  }
  // other strategies
});

Now, to implement returnRangeRequest. This starts with a cache then network approach that you may have seen in a service worker before. The cache is opened and checked against the request. If there is a cached response it is returned and if it is not present in the cache it is fetched from the network, the response cloned and stored in the cache and the result returned.

The important thing is that the result is turned into an ArrayBuffer using Response#arrayBuffer. This will give us access to the raw bytes to build our response from later.

function returnRangeRequest(request, cacheName) {
  return caches
    .open(cacheName)
    .then(function(cache) {
      return cache.match(request.url);
    })
    .then(function(res) {
      if (!res) {
        return fetch(request)
          .then(res => {
            const clonedRes = res.clone();
            return caches
              .open(cacheName)
              .then(cache => cache.put(request, clonedRes))
              .then(() => res);
          })
          .then(res => {
            return res.arrayBuffer();
          });
      }
      return res.arrayBuffer();
    })
    .then(...); // The rest goes here
}

Now we have an arrayBuffer, from either the cache or network response, the real work starts. The Range header looks like:

Range: bytes=200-1000

In this case, the request is for the bytes between the 200th and 1000th byte of the response. The Range header may omit the end byte, meaning that it wants all the bytes from the first value until the end of the file. We can extract these figures with a little regular expression:

  }).then(function(arrayBuffer) {
    const bytes = /^bytes\=(\d+)\-(\d+)?$/g.exec(
      request.headers.get('range')
    );
    // and so on
  });
}

Breaking this regex down quickly, it looks for a string that starts with the "bytes=" followed by some digits a hyphen and optionally some more digits before the end of the string. The two sets of digits are captured, using the brackets, so that we can use them later.

I check to see if the header satisfied the regex and if so, turn the start and end byte values into integers to index into the arrayBuffer. If the end byte index is not present then it is set to the end of the file. With the byte indices the response can be generated.

The body of the response is the slice of the arrayBuffer from the start until the end defined by the Range header. The response carries a 206 status, meaning partial content. It also requires a Content-Range header, which is similar to the original Range header. It tells the browser the response is made of the range of bytes and also returns the total size of the file.

If the regex fails then the service worker will return a 416 error instead.

  }).then(function(arrayBuffer) {
    const bytes = /^bytes\=(\d+)\-(\d+)?$/g.exec(
      request.headers.get('range')
    );
    if (bytes) {
      const start = Number(bytes[1]);
      const end = Number(bytes[2]) || arrayBuffer.byteLength - 1;
      return new Response(arrayBuffer.slice(start, end + 1), {
        status: 206,
        statusText: 'Partial Content',
        headers: [
          ['Content-Range', `bytes ${start}-${end}/${arrayBuffer.byteLength}`]
        ]
      });
    } else {
      return new Response(null, {
        status: 416,
        statusText: 'Range Not Satisfiable',
        headers: [['Content-Range', `*/${arrayBuffer.byteLength}`]]
      });
    }
  });
}

That fixed it. With a reload of the service worker my video started playing in all the browsers again.

Service workers and browsers

You can see the full function in my service worker on GitHub. It's quite a lot of extra work just to get Safari to agree to display video, but it did teach me a bit about what browsers expect when loading video content, how to respond to more intricate requests within a service worker, and how picky Safari can be.

If you have this same problem, I hope my code helps. If you are using Workbox for a service worker it has a range request module that you can use for this too.

If you discovered this problem yourself or if you have another solution for this, I'd love to hear about it. Just drop me a note on Twitter at @philnash.

Techniques for animating on the canvas in React

Thu, 27 Sep 2018 00:00:00 GMT

I recently experimented with audio visualisation in React on the Twilio blog. While I meant to teach myself more about the web audio API I found that I picked up a few techniques for animating in canvas within a React project. If you're creating a canvas animation in React then perhaps this will help you too.

Good references

First up, if you've used React before you'll know that you're supposed to avoid touching the <abbr title="Document Object Model">DOM</abbr> and let React handle it. If you've worked with an HTML5 <canvas> before, you'll also know that to get a context with which to draw on the canvas, you need to call directly on the canvas element itself. Thankfully this is an edge case that React supports through refs.

To get a ref to a canvas element inside a React component you first need to create the ref in the constructor using React.createRef. When you come to render the canvas element, add a prop called ref that points to the ref you created.

class Animation extends React.Component {
  constructor(props) {
    super(props);
    this.canvasRef = React.createRef();
  }

  render() {
    return (
      <div>
        <canvas ref={this.canvasRef} />
      </div>
    );
  }
}

Once you've set it up this way, you can then refer to the canvas element through the ref's current property, for example in componentDidMount:

  componentDidMount() {
    const canvas = this.canvasRef.current;
    const context = canvas.getContext('2d');
    context.fillRect(0, 0, canvas.width, canvas.height);
  }

Now you have the context you can draw and animate all you like.

Separating animation and drawing

A lot of building with React is about maintaining the state of the view. The first time I animated something on a canvas in React I held the state and the code to draw it in the same component. After browsing examples online, I came across this rotating square on CodePen. What I really liked about this example was the way the state was separated from the drawing with the use of two components. The state of the drawing was then passed from the animating component to the drawing component through props.

I recreated the original to show the separation.

First you define a Canvas component that draws an image using the props as parameters.

class Canvas extends React.Component {
  constructor(props) {
    super(props);
    this.canvasRef = React.createRef();
  }

  componentDidUpdate() {
    // Draws a square in the middle of the canvas rotated
    // around the centre by this.props.angle
    const { angle } = this.props;
    const canvas = this.canvasRef.current;
    const ctx = canvas.getContext('2d');
    const width = canvas.width;
    const height = canvas.height;
    ctx.save();
    ctx.beginPath();
    ctx.clearRect(0, 0, width, height);
    ctx.translate(width / 2, height / 2);
    ctx.rotate((angle * Math.PI) / 180);
    ctx.fillStyle = '#4397AC';
    ctx.fillRect(-width / 4, -height / 4, width / 2, height / 2);
    ctx.restore();
  }

  render() {
    return <canvas width="300" height="300" ref={this.canvasRef} />;
  }
}

Then you create an Animation component that runs an animation loop using requestAnimationFrame. Each time the animation loop runs you update the parameters of the animation in the state and let React render the Canvas with the updated props.

Don't forget to implement componentWillUnmount to stop the requestAnimationFrame loop too.

class Animation extends React.Component {
  constructor(props) {
    super(props);
    this.state = { angle: 0 };
    this.updateAnimationState = this.updateAnimationState.bind(this);
  }

  componentDidMount() {
    this.rAF = requestAnimationFrame(this.updateAnimationState);
  }

  updateAnimationState() {
    this.setState(prevState => ({ angle: prevState.angle + 1 }));
    this.rAF = requestAnimationFrame(this.updateAnimationState);
  }

  componentWillUnmount() {
    cancelAnimationFrame(this.rAF);
  }

  render() {
    return <Canvas angle={this.state.angle} />;
  }
}

You can see this in action in this pen.

Rerendering

A concern when animating or doing other intensive visual updates in React is rerendering child elements too often, causing jank. When we are drawing on the canvas we never want the canvas element itself to be rerendered. So what's the best way to hint to React that we don't want that to happen?

You might be thinking of the shouldComponentUpdate lifecycle method. Returning false from shouldComponentUpdate will let React know that this component doesn't need to change. However, if we're using the pattern above, returning false from shouldComponentUpdate will skip running componentDidUpdate and that's responsible for our drawing.

I eventually came across this answer from Dan Abramov to a question on StackOverflow. We can create a PureCanvas component that implements shouldComponentUpdate and returns false and use a callback ref to get the reference to the canvas element in a parent Canvas component.

Note: in Dan's answer he says that using the pattern above should be ok and the following technique is likely only necessary if you have profiled your application and found it makes a difference.

Updating the example above, we split the Canvas component into a Canvas and a PureCanvas. First, the PureCanvas uses a callback ref and a callback provided through the props to return the canvas context to the parent component. It also renders the canvas element itself.

class PureCanvas extends React.Component {
  shouldComponentUpdate() {
    return false;
  }

  render() {
    return (
      <canvas
        width="300"
        height="300"
        ref={node =>
          node ? this.props.contextRef(node.getContext('2d')) : null
        }
      />
    );
  }
}

Then the Canvas component passes a callback function, saveContext, as the contextRef prop when rendering the PureCanvas. When the function is called we save the context (and cache the canvas element's width and height). The rest of the differences from before are turning references to ctx to this.ctx.

class Canvas extends React.Component {
  constructor(props) {
    super(props);
    this.saveContext = this.saveContext.bind(this);
  }

  saveContext(ctx) {
    this.ctx = ctx;
    this.width = this.ctx.canvas.width;
    this.height = this.ctx.canvas.height;
  }

  componentDidUpdate() {
    const { angle } = this.props;
    this.ctx.save();
    this.ctx.beginPath();
    this.ctx.clearRect(0, 0, this.width, this.height);
    this.ctx.translate(this.width / 2, this.height / 2);
    this.ctx.rotate((angle * Math.PI) / 180);
    this.ctx.fillStyle = '#4397AC';
    this.ctx.fillRect(
      -this.width / 4,
      -this.height / 4,
      this.width / 2,
      this.height / 2
    );
    this.ctx.restore();
  }

  render() {
    return <PureCanvas contextRef={this.saveContext} />;
  }
}

Even though it is not necessary, I find this separation between animation, drawing and rendering the canvas element itself quite pleasing. You can see this example in action on CodePen too.

Canvas vs React

It's been an interesting journey working with a canvas element within React. The way they work feels very different to each other, so getting them in sync wasn't necessarily straightforward. Hopefully if you have this problem then these techniques can help you.

If you're interested in other animations in React, do please check out my article on audio visualisation in React.

If you have another way of working with canvas in React I'd love to hear. Drop me a note on Twitter at @philnash.

Implementing one time passwords in Crystal

Tue, 04 Sep 2018 00:00:00 GMT

Crystal is still a young language, there aren't a lot of libraries available yet. For some this could be offputting, but for others this is a chance to learn about a language and provide useful tools for those also starting to use it.

I've given a number of talks over time, some of which were about two factor authentication. A while back I took the opportunity to improve my knowledge and implement the one time password algorithm in Crystal.

This lead me to build the CrOTP library for one time passwords in Crystal.

If you want to check out the code, most of the important implementation can be found in the CrOTP::OTP module. The CrOTP::HOTP and CrOTP::TOTP classes take care of the different ways the counter is treated.

Using CrOTP

To use the CrOTP library in your own Crystal project you need to add it to the dependencies in your shard.yml file.

dependencies:
  crotp:
    github: philnash/crotp

Then install the dependencies with shards install.

Now you can require "crotp" and use it in your application. Most uses on the web of one time passwords are time based tokens for two factor authentication. Here's how to generate and verify time based OTPs.

Getting started

First require the library and generate a random string to use as the secret.

require "crotp"

secret = Random.new.hex(16)

Then create an instance of the CrOTP::TOTP class with that secret.

totp = CrOTP::TOTP.new(secret)

Sharing secrets

You can generate a URI that can be shared with authenticator apps like Authy or Google Authenticator. To do so call authenticator_uri with the name of your app as the issuer and the user account's username as the user.

totp.authenticator_uri(issuer: "Test app", user: "philnash@example.com")

Turn the resulting URI into a QR code and users can scan it to add the account to their app.

Generating and verifying

Now you can use the totp object to generate a new one time password that will be valid within the current 30 second period of time.

password = totp.generate

To verify a code, you can use the same object, calling verify instead.

totp.verify(password)

You can also allow a drift, to give users more time to enter their code. The drift is the number of periods (periods are 30 seconds long by default) that can pass after the token was generated.

totp.verify(password, at: Time.now, allowed_drift: 2)

There are more examples in the project repo.

There's more work to be done

CrOTP does the basic work for generating and verifying one time passwords. Of course, there's more to do on the project when I find the time.

I'd love to hear if this project is useful to you or if you're interested in helping implement the missing features. Let me know on Twitter at @philnash.

Git commands to keep a fork up to date

Tue, 21 Aug 2018 00:00:00 GMT

I've seen the following tweet about git making its way around Twitter recently:

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">I ❤️ Git. But honestly, it intimidated me for years. I thought I needed to understand all its powerful features to be productive. I found that's not the case. You can be quite productive in Git with around 6 commands: <br><br>branch<br>checkout<br>add<br>commit<br>pull<br>push</p>— Cory House 🏠 (@housecor) <a href="https://twitter.com/housecor/status/1031190760278970368?ref_src=twsrc%5Etfw">August 19, 2018</a></blockquote>

And it's true, you can do most of your work with those commands.

What if you want to fork and contribute to an open source project on GitHub, GitLab or BitBucket though? You're going to need a few more commands so that you can keep your fork up to date, namely remote, fetch and merge. Let's see how they are used.

Fork away

When you fork a project, you make a copy of it under your own namespace. To work with the project you then clone the repository to your own machine. Let's use a repo I have forked as an example: Twilio's Node.js package. After forking, we clone the repo:

git clone git@github.com:philnash/twilio-node.git

We can now see the first use case for remote. Run git remote in the twilio-node directory and we see the following:

git remote
origin

OK, that's not too useful, it just shows we have one remote repo called "origin". Running it again with the --verbose flag shows a bit more information:

git remote --verbose
origin	git@github.com:philnash/twilio-node.git (fetch)
origin	git@github.com:philnash/twilio-node.git (push)

That's better, this shows that the remote repository is the one we cloned and that we can fetch from and push to it.

Making changes

When contributing something back to the repo, it is easiest to do so on a branch. That keeps the master branch clean and makes it easy to keep up to date. Setting up your changes is covered by the six commands Cory listed in his tweet; you create a new branch with branch, checkout the branch, make the changes you want, commit as many times as necessary and finally push the branch to the origin, your fork. Then you can create your pull request and hope it gets accepted.

Sometime later down the line you find you want to make another pull request, but the original repo has moved on. You need to update your fork so that you're working with the latest code. You could delete your fork and go through the process of forking and cloning the repo again, but that's a lot of unnecessary work. Instead, add the upstream repository as another remote for the repo and work with it from the command line.

Adding the upstream

To do this, we use the remote command again, this time to add a new remote. Find the original repo's URL and add it as a new remote. By convention this remote is called "upstream" though you can call it whatever you want.

git remote add upstream git@github.com:twilio/twilio-node.git

Now if we inspect the repo's remotes again, we will see both the origin and the upstream. You can use -v as a shortcut for --verbose.

git remote -v
origin	  git@github.com:philnash/twilio-node.git (fetch)
origin	  git@github.com:philnash/twilio-node.git (push)
upstream	git@github.com:twilio/twilio-node.git (fetch)
upstream	git@github.com:twilio/twilio-node.git (push)

Fetching the latest

To bring the repo up to date, we can now fetch the latest from the upstream repo. It looks like this:

git fetch upstream
remote: Counting objects: 6427, done.
remote: Compressing objects: 100% (549/549), done.
remote: Total 6427 (delta 5156), reused 5105 (delta 4939), pack-reused 934
Receiving objects: 100% (6427/6427), 2.54 MiB | 1.28 MiB/s, done.
Resolving deltas: 100% (5399/5399), completed with 391 local objects.
From github.com:twilio/twilio-node
   6a6733a8..73656c50  master           -> upstream/master

git fetch downloads objects and refs from the repository, but doesn't apply it to the branch we are working on. We want to apply the updates to the master branch, so make sure it is checked out.

git checkout master

Merging it all together

To bring the master branch up to date with the remote merge the remote's master branch into our own, like so:

git merge upstream/master

If you have kept your work off the master branch this will go ahead smoothly. Finally push these updates to the fork so that it is up to date too.

git push origin master

Now everything on the master branch of the fork is up to date. If you need to update a different branch, substitute the branch name master for the branch you are working with.

Shortcuts

If you have been working with git pull then you may have already seen a potential shortcut. pull is the combination of fetch and merge, so to perform these two actions in one command you can run:

git pull upstream master

Fork happy

That's all you need to know for keeping your fork up to date when contributing to open source. Add the upstream as a new remote repo, fetch the upstream repo and merge the branch you want to update.

For more detail on the various commands here, take a look at "Working with Remotes" from the Pro Git book.

And remember, if you get stuck with something with git, check out Oh shit, git! for all your expletive based ways to save yourself.

Happy forking!

Spring clean your dev machine

Sun, 27 May 2018 00:00:00 GMT

Development machines can build up such a lot of cruft. Old versions, oudated programs and unused caches litter the hard drive. It's good to take time once in a while to clean all of this up and free up some space.

Here are some tips for commands you can run or actions you can take to clean up your machine. If you have a tip that I'm missing here, please share it with me on Twitter.

Homebrew

If you're using Homebrew to manage packages on macOS you can run brew cleanup to remove old versions of packages and old downloads from the cache.

$ brew cleanup
Removing: ...
...
==> This operation has freed approximately 6.9GB of disk space.

Using the -s flag scrubs the downloads for the latest package versions from the cache to give you back even more space. Thanks to David Guyon for that tip.

To really turn it up to the max, Daniel Miller suggested a bash alias to update Homebrew, upgrade packages and then cleanup after yourself. Add the following to your .bash_profile:

alias bu="brew update && brew upgrade && brew cleanup"

Then run the commands with:

bu

Everything should be up to date and leave no mess behind!

Homebrew bonus

Once all the caches have been tidied up, take a moment to make sure Homebrew itself is running smoothly. Run the following command for a list of actions you can take to tidy up your install.

brew doctor

Docker

A few gigabytes of packages is a pretty good, but can we do better? If you're using Docker you can clean things up with:

$ docker volume prune
...
Total reclaimed space: 40.77GB

I am not a big Docker user myself, this was a tip from Jack Wearden.

Rubies

I install multiple versions of Ruby using rbenv and ruby-build. I just checked the versions I have installed and I found 12 Rubies that are beyond end of life. Since they also have their gems installed alongside them, clearing them out saved me tens to hundreds of megabytes per Ruby version.

You can check the versions you have installed with:

rbenv versions

You can then uninstall an unwanted version of Ruby with:

rbenv uninstall 2.1.0

Nodes

I also use nvm to manage multiple versions of Node.js. The drill is the same here as with Ruby. Find the old Node versions that you have hanging around with:

nvm ls

Then uninstall with:

nvm uninstall v6.9.2

Any other ideas?

That should clear up a bunch of space on your machine, it sure did on mine. Now you have more room for more installs, more containers and more versions of more languages!

I'd love to collect any other tips that you might have to keep a development machine running smooth and lean. Let me know how you spring clean your development machine on Twitter at @philnash.

<footer> <small>Dust icon by <a href="https://thenounproject.com/smalllike/collection/cleaning/?i=1683969">Smalllike from the Noun Project</a></small> </footer>

Use the web share API easily with web components

Wed, 25 Apr 2018 00:00:00 GMT

I'm a fan of the web share API but I was not happy with my initial implementation of the API. It was all a bit complex for what is a very simple API. I wanted something more declarative and easier to use with a more generic fallback for when the share API was not available.

I've been playing about with web components on and off over the last year and I wanted to see if I could make a component that would make using the web share API really easy. I think I've done it and I hope you like it.

The `<web-share-wrapper>`

I've built a web component called <web-share-wrapper>. It is a wrapper because it is intended to wrap around any existing elements that you might already be using as share buttons to various social networks and replace them with a single element that uses the web share API.

If you are using Chrome on Android (the only current browser that supports the web share API) then you can check out the <web-share-wrapper> in action. Either scroll to the bottom of this article and click the share link or take a look at the examples in the repo.

Using the `<web-share-wrapper>`

There are instructions for using the <web-share-wrapper> in the readme on GitHub but here's a quick example.

<web-share-wrapper text="Share" sharetext="Check out the web-share-wrapper web component" shareurl="https://github.com/philnash/web-share-wrapper">
  <a href="https://twitter.com/intent/tweet/?text=Check%20out%20%40philnash's%20web-share-wrapper%20web%20component&amp;url=https%3A%2F%2Fgithub.com%2Fphilnash%2Fweb-share-wrapper">Share on Twitter</a>
</web-share-wrapper>

If the web share API is not available, the above HTML will just be a link to share the project via a Twitter tweet intent. If the web share API is available, then the web component will kick in and replace the link with a button that, when clicked, will invoke the web share API. The button text is supplied by the text attribute, the sharetext attribute sets the body of the share and the shareurl sets the link to share.

The button is injected into the regular DOM, not the shadow DOM, so you can style it using regular CSS.

Getting fancy with templates

If a plain button isn't your style, the <web-share-wrapper> also supports using HTML templates. Instead of using the text attribute to set the button text, use a template attribute to point to the id of a <template> on the page and the <web-share-wrapper> will hydrate that <template> and insert it in place of the content.

For example, the following code will use an <a> instead of a <button>:

<template id="link">
  <a>Share link</a>
</template>

<web-share-wrapper template="link" sharetext="Check out the web-share-wrapper web component" shareurl="https://github.com/philnash/web-share-wrapper">
  <a href="https://twitter.com/intent/tweet/?text=Check%20out%20%40philnash's%20web-share-wrapper%20web%20component&amp;url=https%3A%2F%2Fgithub.com%2Fphilnash%2Fweb-share-wrapper">Share on Twitter</a>
</web-share-wrapper>

Again, you can style the contents of the template anyway you like. The examples in the repo show a fancier button with an icon.

Installing `<web-share-wrapper>`

The web component is published to Bower and you can install it to a project with:

bower install --save web-share-wrapper

I am investigating other ways of publishing this web component as, while webcomponents.org seems to favour publishing to Bower, this doesn't seem like the best long term plan for a project. Support for npm seems like the most important next step.

What do you think?

You can now use the web share API declaratively, with just a single web component. I'd love to know your thoughts on how this has been implemented and whether you think it's a good idea. Let me know on Twitter at @philnash or in the issues on the GitHub repo.

CSS: select first-of-class with the subsequent sibling combinator

Sun, 18 Mar 2018 00:00:00 GMT

There are a whole bunch of CSS selectors available to web developers, but sometimes there's still not enough. I found this recently when building the speaking section of my site and wanted to use the non-existent :first-of-class pseudo class to apply some styles.

The problem

The pseudo class :first-of-type does exist, but it is limited. It is a special case of the :nth-of-type() pseudo class. From the spec (emphasis mine):

The :nth-of-type(An+B) pseudo-class notation represents the same elements that would be matched by :nth-child(|An+B| of S), where S is a type selector and namespace prefix matching the element in question.

That means, when using :nth-of-type() or :first-of-type the selector can only be a type selector. That is, you can only refer directly to elements—like <p> or <h1>—when using :first-of-type. So, if you had the following HTML:

<div>
  <p>This is paragraph 1</p>
  <p class="special">This is paragraph 2</p>
  <p class="special">This is paragraph 3</p>
</div>

And this CSS:

div p { color: #333; }
div p:first-of-type { color: red; }
div p.special:first-of-type { color: green; }

You might expect paragraph 1 to be coloured red and paragraph 2 to be coloured green. But div p.special is not a type selector, so :first-of-type doesn't apply here and nothing is green.

The subsequent sibling combinator

This can be solved with a technique created by Daniel Tan and Lea Verou and shared on Stack Overflow. Rather than using pseudo classes at all, we can use the subsequent or general sibling combinator.

The subsequent sibling combinator looks like A ~ B where A and B are two compound selectors (not just type selectors). It allows you to select B where A and B share a parent and A precedes B in the document.

With the following HTML, the selector h2 ~ p would select the second <p> as it follows the <h2> but not the first <p>.

<div>
  <h1>Heading</h1>
  <p>Meta data</p>
  <h2>Subheading</h2>
  <p>Some article text</p>
</div>

The subsequent sibling combinator and :first-of-class

The trick to using the combinator to emulate a :first-of-class psuedo class is to use a regular selector to style all the elements of the class with the style you want. Then use the combinator to turn it off for all but the first element. In our original example, the CSS now looks like this:

div p { color: #333; }
div p:first-of-type { color: red; }
div p.special { color: green; }
div p.special ~ p.special { color: #333; }

Now the first paragraph will be red, the second green and the last one grey. You can check out a live example of this on Codepen.

CSS Selectors level 4

After posting this, Šime Vidas pointed out on Twitter that the future of CSS holds more promise for this style of selector. In the CSS Selectors level 4 specification the :nth-child() pseudo class takes an argument that looks like: An+B [of S]?. The An+B part means you can provide a function to calculate what n is, but the optional of S means the pseudo class will match the Nth element that matches the selector.

This means we can update our example to use :nth-child() instead of two rules like we did above. Check out the CSS below:

div p { color: #333; }
div p:first-of-type { color: red; }
div :nth-child(1 of p.special) { color: green; }

Using :nth-child(1 of p.special) means we are selecting the first child of the <div> that is a <p> with a class of "special". This is exactly what I wanted.

The only drawback with this technique? It only works in Safari right now. I've updated the Codepen, check it out in Safari to see all your future selector dreams come true.

CSS hacking is still fun

I've been writing CSS on and off for more than a decade now and while all the modern layout capabilities of CSS mean that there's a lot less hacking, sometimes you need to come up with a creative solution to a problem. I could have solved this with an extra class or a rearrangement of my HTML, but when CSS can do the job for you it feels more satisfying.

Thanks to Daniel and Lea for sharing their solutions on Stack Overflow and particularly to Daniel who's answer goes into a lot more detail about CSS selector and pseudo class misunderstandings.

Maybe one day we'll see a :first-of-class pseudo class. With the latest version of :nth-child() maybe we don't even need a :first-of-class, we're just waiting for browser support. In the meantime the subsequent sibling combinator remains our friend.

gzip a file in Ruby

Sun, 25 Feb 2018 00:00:00 GMT

At the start of the year I looked into how to better compress the output of a Jekyll site. I'll write up the results to that soon. For now, here's how to gzip a file using Ruby.

Enter zlib

Contained within the Ruby standard library is the Zlib module which gives access to the underlying zlib library. It is used to read and write files in gzip format. Here's a small program to read in a file, compress it and save it as a gzip file.

require "zlib"

def compress_file(file_name)
  zipped = "#{file_name}.gz"

  Zlib::GzipWriter.open(zipped) do |gz|
    gz.write IO.binread(file_name)
  end
end

You can use any IO or IO-like object with Zlib::GzipWriter.

We can enhance this by adding the files original name into the compressed output. Also, it can be beneficial to set the file's modified time to the same as the original, particularly if you are using the file with nginx's gzip_static module.

require "zlib"

def compress_file(file_name)
  zipped = "#{file_name}.gz"

  Zlib::GzipWriter.open(zipped) do |gz|
    gz.mtime = File.mtime(file_name)
    gz.orig_name = file_name
    gz.write IO.binread(file_name)
  end
end

Just to see how well this is working, we can print some statistics from the compression.

require "zlib"

def print_stats(file_name, zipped)
  original_size = File.size(file_name)
  zipped_size = File.size(zipped)
  percentage_difference = ((original_size - zipped_size).to_f/original_size)*100
  puts "#{file_name}: #{original_size}"
  puts "#{zipped}: #{zipped_size}"
  puts "difference - #{'%.2f' % percentage_difference}%"
end

def compress_file(file_name)
  zipped = "#{file_name}.gz"

  Zlib::GzipWriter.open(zipped) do |gz|
    gz.mtime = File.mtime(file_name)
    gz.orig_name = file_name
    gz.write IO.binread(file_name)
  end

  print_stats(file_name, zipped)
end

Now you can pick a file and compress it with Ruby and gzip. For this quick test, I chose the uncompressed, unminified jQuery version 3.3.1. The results were:

./jquery.js: 271751
./jquery.js.gz: 80669
difference - 70.32%

Pretty good, but we can squeeze more out of it if we try a little harder.

Compression levels

Zlib::GzipWriter takes a second argument to open which is the level of compression zlib applies. 0 is no compression and 9 is the best possible compression and there's a trade off as you progress from 0 to 9 between speed and compression. The default compression is a good compromise between speed and compression, but if time isn't a worry for you then you might as well make the file as small as possible, especially if you want to serve it over the web. To set the compression level to 9 you can just use the integer, but Zlib has a convenient constant for it: Zlib::BEST_COMPRESSION.

Changing the line with Zlib::GzipWriter in it to:

  Zlib::GzipWriter.open(zipped, Zlib::BEST_COMPRESSION) do |gz|

and running the file against jQuery again gives you:

./jquery.js: 271751
./jquery.js.gz: 80268
difference - 70.46%

A difference of 0.14 percentage points! Ok, not a huge win, but if the time to generate the file doesn't matter then you might as well. And the difference is greater on even larger files.

Streaming gzip

There's one last thing you might want to add. If you are compressing really large files, loading them entirely into memory isn't the most efficient way to work. Gzip is a streaming format though, so you can write chunks at a time to it. In this case, you just need to read the file you are compressing incrementally and write the chunks to the GzipWriter. Here's what it would look like to read the file in chunks of 16kb:

def compress_file(file_name)
  zipped = "#{file_name}.gz"

  Zlib::GzipWriter.open(zipped, Zlib::BEST_COMPRESSION) do |gz|
    gz.mtime = File.mtime(file_name)
    gz.orig_name = file_name
    File.open(file_name) do |file|
      while chunk = file.read(16*1024) do
        gz.write(chunk)
      end
    end
  end
end

gzip in Ruby

So that is how to gzip a file using Ruby and zlib, as well as how to add extra information and control the compression level to balance speed against final filesize. All of this went into a gem I created recently to use maximum gzip compression on the output of a Jekyll site. The gem is called jekyll-gzip and I'll be writing more about it, as well as other tools that are better than the zlib implementation of gzip, soon.

<footer> <small>Header icons: <a href="https://thenounproject.com/term/diamond/315/">Diamond by Edward Boatman</a> and <a href="https://thenounproject.com/search/?q=vice&i=1554537">vice by Daniel Luft</a> from the Noun Project.</small> </footer>

Permissions on the web suck

Mon, 08 Jan 2018 00:00:00 GMT

I am a fan of progressive web apps and the powers that they bestow on web developers to build the next generation of applications. We can write web applications that work offline, download large files in the background, send push notifications, and much more. I was so excited about push notifications on the web that I wrote a whole talk about it in 2015 and was fortunate enough to give it in a bunch of places around the world.

But perhaps I was a little too prescient with that talk title, "The web is getting pushy." Web applications themselves are getting pushy and now I see tweets like this:

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Dear Every Single Website in Existence,<br><br>No, I do not want Notifications turned on.<br><br>Thanks,<br><br>Dan</p>— Dan Szymborski (@DSzymborski) <a href="https://twitter.com/DSzymborski/status/921260402146672641?ref_src=twsrc%5Etfw">October 20, 2017</a></blockquote>

And this:

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Dear websites: no one wants push notifications. Love, every person who uses the internet</p>— Andrea Whitmer (@nutsandbolts) <a href="https://twitter.com/nutsandbolts/status/925421346259193862?ref_src=twsrc%5Etfw">October 31, 2017</a></blockquote>

And blog posts like How-To Geek's how to stop websites from asking to show notifications.

Push notifications are getting a bad reputation and I don't think they deserve it. Here are the problems I think we are facing with this and some potential solutions.

It's not the fault of the notifications

I have a theory. It's not that users don't want push notifications. There is a time and a place for a good push notification. Native mobile application developers seem to be getting this right now, at least in my experience, and innovative web teams like those at the Guardian have done some really interesting and impressive push notification experiments.

My theory is that users might want push notifications. They might want them for newsworthy moments, like the Guardian's election night experiments. They might want notifications that someone has sent them a message or that their taxi is arriving or their flight has been delayed. There are countless reasons a user might want to receive push notifications.

But the top way to annoy any user is to pop up that permission dialog asking to send push notifications on page load without any context, any information at all, that would allow them to make that decision.

I noticed Sitepoint, a web tutorial site that really should know better, doing this. And other well known sites; Product Hunt, cnet and even Facebook in their early experimentation with the feature, have been spotted doing it too. There are probably many more examples.

These permission dialogs suck.

Permission for what?

Read that dialog from the screenshot again. All it says is "www.sitepoint.com wants to show notifications" and there are two buttons, "Block" or "Allow". It doesn't say what the notifications will contain, how often they might be sent, why the user should even care. That permission dialog can't say that. There is nothing in the PushManager API that can be used to add any context to a popup like this.

I believe that the intention for the API is to encourage developers to build an intermediary step where the application explains why it wants permission to send push notifications. Then when the user agrees to that, trigger the real permission notification.

This is a pattern that Matt Gaunt explains beautifully with his airline example from his article on permissions UX. The real key to each of the patterns in Matt's article is that the permission dialog never surprises the user, they always know why they are being asked permission by the browser to send notifications.

Permissions solved

If everyone just reads Matt's article and implements friendly patterns for asking for permission then everything is solved, right? If only.

It doesn't matter how good an article on UX is or how many people read it, it can't reach everyone. So we still end up with permissions popping up at page load. You might think that this is just bad for the site that is providing the poor experience, but check those tweets at the start of this post. They don't care any more, they never want notifications. They want to be able to turn them all off for good. And they can, that's what the How-To Geek article explains. Firefox is soon releasing a global disable option too, and if you read the responses you'll see that this is being welcomed.

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">There is now a global "disable" option for all push notification prompts<a href="https://t.co/b01ZCoWOfK">https://t.co/b01ZCoWOfK</a> <a href="https://t.co/0JSSYoaysb">pic.twitter.com/0JSSYoaysb</a></p>— Firefox Nightly (@FirefoxNightly) <a href="https://twitter.com/FirefoxNightly/status/950321939129815040?ref_src=twsrc%5Etfw">January 8, 2018</a></blockquote>

The problem here is that all these poor experiences with permissions are causing users to revoke the permission not just from the offending sites but from the entire platform. Once notifications are turned off globally, it's almost impossible to get users to turn them back on again. Now, even if you've built the best push notification permission flow, a user who has outright blocked notifications will never see it and never experience your application the way you built it.

I think we need more than just best practice UX articles to solve this.

Power to the browser

I believe the power is in the hands of the browsers. We've seen them deal with unnecessary popups before. Remember, back in the day, installing popup blockers because the web was a fraught mess of windows popping up everywhere. Now every browser has a built in popup blocker. Most of what these built in popup blockers do is restrict the window.open function to only work on user interaction. This means that users browsing the web don't end up with a whole bunch of unexpected popups or worse, popunders.

I would like to see the same for permission dialogs. If the browsers enforced a user interaction before you could show a permission dialog then the page load permission dialog would disappear immediately. The platform would then encourage all developers to explain the permission before asking for it and lead to better experiences all round.

Together we can save permissions

It's going to be a team effort, but I think we can save permissions dialogs, push notifications and the web platform.

<strong>Developers</strong>: never show a permissions dialog on page load, instead read Matt Gaunt's article on permission UX and build better experiences.

<strong>Browsers</strong>: please consider a permissions popup blocker and encourage developers to build better applications by making poor experiences harder to build.

<strong>Users</strong>: don't block all permissions, please, you might be missing out on something really useful somewhere else on the web.

I hope we can all agree that there are uses for push notifications, what we really need to fix are the permissions.

Two tests you should run against your Ruby project now

Wed, 12 Jul 2017 00:00:00 GMT

I had the fortune of attending the wonderful Brighton Ruby conference last week. It was full of excellent advice, wisdom and code. There was one talk, however, that urged me to go home and do something straight away to ensure the security and usability of my projects.

Andrew Nesbitt, with the backing of the libraries.io open data, told tales of the hidden costs of open source software in Ruby and beyond. His slides are available here and I will link to the video when it is published.

Counting the costs

There were two take aways from Andrew's talk that could be acted on immediately: check up on security vulnerabilities and license issues in my projects.

Security audits

Rubysec maintains the Ruby Advisory Database, a database of vulnerable Ruby gems. The database currently contains 287 advisories across 147 gems. If you're worried that your project depends on a vulnerable gem they publish bundler-audit, which is a gem that audits Gemfile.locks against the their database.

To use bundler-audit you need to install it:

$ gem install bundler-audit

and run it in your project directory.

$ bundle-audit
No vulnerabilities found

Second opinions

If one security audit isn't enough to quell your fears then you are in luck. While it isn't a Ruby tool and Andrew didn't mention it, Snyk is a tool for monitoring vulnerabilities in Ruby, JavaScript and Java projects via their own vulnerability database.

They publish a command line interface that you can use to check your applications. You do need to create an account to use it and install Node.js and npm to run it. Once over those hurdles you can install and run it against your project like so:

# install
$ npm install -g snyk
# authenticate the client
$ snyk auth
# test the project in the current directory
$ snyk test
✓ Tested 23 dependencies for known vulnerabilities, no vulnerable paths found.

Now security has been checked, how about licenses?

Licensing audits

Unlicensed code is copyrighted code that you do not have the permission to use.

This may not sound bad until you hear that 28% of all Ruby gems have no license declared. That is over 37,000 gems. And you don't need to worry about just the gems that you install, but their dependencies. And their dependencies. And their... it goes on.

Thankfully there is a tool to check this too. license_finder is maintained by Pivotal and checks your dependencies for licenses. The tool reports back with each dependency's license. You then have to decide whether the license fits with your project. You can choose to accept each of your dependencies and you can bulk accept by whitelisting certain licenses. If you're not using Ruby, license_finder works with many types of project.

You install license_finder with Ruby gems.

$ gem install license_finder

Then you run it within a project directory. This was the result when I ran it against envyable:

$ license_finder
..............................................................................
Dependencies that need approval:
bundler, 1.12.5, MIT
codeclimate-test-reporter, 1.0.8, MIT
docile, 1.1.5, MIT
envyable, 1.2.0, MIT
minitest, 5.10.2, MIT
rake, 12.0.0, MIT
simplecov, 0.13.0, MIT
simplecov-html, 0.10.1, MIT
thor, 0.19.4, MIT

Then, when I whitelist the MIT license and run it again:

$ license_finder whitelist add MIT
$ license_finder
..............................................................................
All dependencies are approved for use

The licenses are approved.

Test all the time

Whether there is a security vulnerability or an unlicensed project you could be subjecting yourself or your users to problems.

It doesn't take long to install and run these tools and I encourage you to do so too, they may save you from your dependencies. It takes a little longer, but it's probably worth it, to add them to your CI setup too.

To find out more look out for the video of Andrew's talk, check out libraries.io's list of libraries without licenses or even take a dive into the data and see what you can find out about our open source ecosystem.

Experimenting with the background fetch API

Tue, 04 Jul 2017 00:00:00 GMT

The service worker API is expanding as more ways to use the background dwelling worker emerge. I've written before about push notifications and background sync and I recently explored the very new background fetch API. Here's what I found out about it.

Downloads and uploads

The background fetch API seeks to solve two problem:

When a service worker is downloading large files to cache and the user navigates away, even when using event.waitUntil the service worker may eventually be killed
When uploading files and the user navigates away the upload is interrupted and will fail

The background fetch API allows a developer to perform and control fetches to large files or lists of files outside the context of a single page. This can increase the likelihood of successful uploads and downloads and allow the service worker to cache the results too.

This API is available to test, at the time of writing, in Chrome with the experimental web platform features flag turned on in the chrome://flags settings.

Testing out an example

To try to understand how the API worked, I decided to build a simple proof of concept. The example can be found here on Glitch or you can check out and run the source code from GitHub. I didn't want to mess around with large files, so this example tries to load an image that has a 10 second delay on it (4.5 seconds on Glitch).

There is a service worker set up to serve the image directly from the cache if it is present, otherwise it goes to the network.

// sw.js
self.addEventListener('fetch', function(event) {
  if (event.request.url.match(/images/)) {
    event.respondWith(
      caches.open('downloads').then(cache => {
        return cache.match(event.request).then(response => {
          return response || fetch(event.request);
        });
      })
    );
  } else {
    event.respondWith(caches.match(event.request));
  }
});

Pretty standard service worker stuff so far, here's the new stuff.

Clicking on the "Start download" button will kick off a background fetch. The fetch is tagged 'large-file' (I was feeling creative). While the background fetch is going on, you can navigate away from the page, the download will continue to work in the background.

// index.html
if ('serviceWorker' in navigator) {
  navigator.serviceWorker.register('/sw.js').then(function(reg) {
    const button = document.getElementById('download');
    if ('backgroundFetch' in reg) {
      button.addEventListener('click', function(event) {
        reg.backgroundFetch.fetch('large-file', [new Request('/images/twilio.png')], {
          title: 'Downloading large image'
        }).then(function(backgroundFetch) { console.log(backgroundFetch) });
      })
    }
  });
}

When the download is complete the service worker receives the backgroundfetched event. The event has a fetches property that points to an array of objects containing each request and response. The code loops through the fetches (even though in this case we know there's only one) and puts the response into the cache.

self.addEventListener('backgroundfetched', function(event) {
  event.waitUntil(
    caches.open('downloads').then(function(cache) {
      event.updateUI('Large file downloaded');
      registration.showNotification('File downloaded!');
      const promises = event.fetches.map(({ request, response }) => {
        if (response && response.ok) {
          return cache.put(request, response.clone());
        }
      });
      return Promise.all(promises);
    })
  );
});

Now, when the image is requested it will be served directly from the service worker cache, no more slow download.

Thoughts on the background fetch API

As you can see, it's not too hard to implement the background fetch API. There are some events I didn't implement on this initial exploration, the backgroundfetchfail, backgroundfetchabort and backgroundfetchclick events. In a fully featured application you would expect to do so.

Audio difficulties

I initially started this experiment with an audio file, but found it hard to prove I had stored the audio file successfully in the cache. I don't blame the background fetch API for this, rather the issues in dealing with range requests in the service worker fetch event. Given that this API is very useful for downloading large audio and video assets for rich applications, making this easier for developers should be an important part of service worker discussions for anyone working on this feature.

Popping up in the downloads

When testing this in Chrome on the desktop I was surprised to find that the file was downloaded in a user visible manner. My expectations were that the file would be left for the service worker to handle and cache. Instead it appeared in my downloads folder. If you were using the background fetch API to fetch resources for a game, for example, I would not expect to litter the user's downloads folder with resources that are only of use within the application. This didn't happen when I tested on an Android device though.

What UI?

Within the API there are places to affect the browser UI, you can set a title and icons for the fetch and update the title using the event.updateUI method. However, I couldn't see this UI anywhere in my testing on desktop or mobile. I can only assume that this is being worked on and will arrive with another version of Canary.

To the future

I like to play with these early APIs, there's such a lot of potential in them (I'm excited for the return of the Web Share API too!). The background fetch API is going to be great for applications that require downloading large files or large amounts of files, with more control than just using the service worker cache. It's obviously going to be useful for video and audio applications, but games and virtual reality experiences will surely benefit from it too.

I think there is still work to be done in browser to make this experience a good one for both developers and users, but it's exciting to see how the service worker is progressing. My next plan is to build an application that fully takes advantage of the background fetch API, as well as other service worker features, in a real world situation. Keep an eye on my GitHub profile to follow the progress.

If you're interested in the background fetch API there are more examples and discussion on the proposal on GitHub. Give me a shout on Twitter if you're also excited about features like this coming to the browser.

Speed up bundle install with this one trick

Mon, 12 Jun 2017 00:00:00 GMT

Did you know bundler can download and install gems in parallel?

That's right, using the --jobs option for bundle install means you can set the number of gems that can download and install at the same time. To use four jobs, for example, you can run:

bundle install --jobs=4

As a bonus, you don't even have to remember to use the --jobs option every time. You can set this in your global bundler config with this command:

bundle config --global jobs 4

Does it really help?

I found all of this out recently, even though this has been available since bundler version 1.4.0, and I wanted to test how useful it actually was. Downloading and installing gems in parallel sounds like a delight, but I needed to test it with something to be sure.

I went looking for a large Rails project to try this out on. I happened upon the GitLab Community Edition app and cloned the project. I didn't need to actually get the project running, I was just testing the installation time of the gems.

I was looking for a large application. rake stats told me that it contains 172 controllers, 208 models and 86,019 lines of code. In the Gemfile there are 197 gem dependencies which shakes out to 369 gems in total. I checked out Discourse and Diaspora too, but GitLab certainly had the largest number of gems, so it was perfect to test my theory.

I ran time bundle install --path=./gems --quiet --force --jobs=n five times each for n equal to 1 and 4. The median results for each were:

time bundle install --path=./gems --quiet --force --jobs=1
real  4m39.836s
user  1m59.692s
sys 0m50.291s

time bundle install --path=./gems --quiet --force --jobs=4
real  2m55.857s
user  2m0.005s
sys 0m47.897s

These tests were run on a MacBook Pro with 2.5 GHz Intel Core i7 and 16GB RAM.

With these results we can see that installing the gems for GitLab in parallel using 4 workers was ~1.6 times faster.

You should run your own tests with your own setup to see if this genuinely will save you time with installing gems. There has been some research into the best number of jobs. I've certainly set the default number of jobs to 4 for the future.

Why isn't this default?

You might be wondering why Bundler doesn't just turn on parallel installation by default. A quick glance at the comments in the source code gives this one away.

# the order that the resolver provides is significant, since
# dependencies might affect the installation of a gem.
# that said, it's a rare situation (other than rake), and parallel
# installation is SO MUCH FASTER. so we let people opt in.

Reading the docs

As a final point, I'd like to point out that reading the documentation, even for a project you may use every day, can turn up some interesting options you didn't know you could use. Bundler itself is full of other useful commands and options, check out bundle pristine, bundle gem and bundle outdated for some ideas of what is available.

So, keep an eye out for interesting options in the software you use and go out there and install your gems faster.

Always install Bundler alongside Ruby with rbenv

Wed, 22 Mar 2017 00:00:00 GMT

Here's a quick tip for anyone who uses rbenv and ruby-build to manage installing and switching Ruby versions.

When installing a new version of Ruby I cannot think of a situation in which I wouldn't want Bundler installed too. The good news is that you can get rbenv to install Bundler, or any other gem you like, as soon as you install a new version of Ruby. All you need is the default-gems plugin.

Default gems

If you installed rbenv with Homebrew, like me, then you can install default-gems with Homebrew too.

$ brew install rbenv-default-gems

Then you need to make a list of gems that you want installed. First create a file named default-gems in your $(rbenv root) directory. Mine is located at ~/.rbenv/default-gems. Add the names of the gems you want to install, one per line. You can include an optional version or the option --pre for a prerelease version. For Bundler you will want the latest release, so the default-gems file would look like this:

bundler

Now when you install a new version of Ruby, Bundler will be installed as well.

You can add other gems that you always use too. If you're a fan of pry, for example, then add a line for it in default-gems and you'll never have to remember to install it yourself.

rbenv plugins

There are a number of rbenv plugins which you can check out, but default-gems has always been a must have for me.

Are there any tools or plugins you use that make your Ruby development easier? I'd love to hear about them! Please share your tips with me on Twitter.

The web share API

Tue, 14 Mar 2017 00:00:00 GMT

Recently I implemented the web share API for my site as a means of testing it out. If you are using version 55 or above of Chrome on Android then you can see it in action by clicking "share it" at the bottom of this post.

If you don't have a browser that supports the API, then this is what it looks like when you hit the share link.

<div class="info"> <p>The Web Share API origin trial has finished and after consideration is now supported in Chrome for Android. You can keep up with support by checking out the <a href="https://caniuse.com/#search=web-share">Web Share API on Can I Use</a>.</p> </div>

I want to share how the API works and my thoughts on it so far.

How it works

The web share API is actually quite a simple API. It is available on the navigator object and consists of one method; share. The method takes an object that can have title, text or url properties. It is mandatory to have either a text or url property, though you can have both.

You can only invoke navigator.share as a result of a user gesture, so sites can't surprise a user into sharing a page. Also, like other powerful new web platform features, it only works on sites hosted on HTTPS.

The API is Promise based, when a page is successfully shared the Promise will resolve. If the user is unable to or chooses not to share, the Promise will reject.

You can feature detect the existence of the API so you can progressively enhance a site with the web share API. This is good news right now as the API is only available as an Origin Trial. This means that it is not widely available and you need to opt in to testing the feature on your site. If you are on a browser that doesn't yet support the API and you click "share it" at the bottom of this post, you just get directed to the Twitter web intent.

Implementing the web share API

This is how you'd call the web share API on its own.

navigator.share({ title: title, url: url })
  .then(function() { console.log("Share success!"); })
  .catch(function() { console.log("Share failure!"); });

And here is my first implementation of the web share API for this site.

var shareLinks = [].slice.call(document.querySelectorAll('.share'));
shareLinks.forEach(function(link) {
  link.addEventListener('click', function (event) {
    if (typeof navigator.share !== 'undefined') {
      event.preventDefault();
      var canonicalElement = document.querySelector('link[rel=canonical]');
      if(canonicalElement !== undefined) {
        var url = canonicalElement.href;
      } else {
        var url = window.location.href;
      }
      var pageTitle = document.querySelector('.post-title').textContent;
      navigator.share({ title: pageTitle, url: url })
        .then(function() { console.log("Share success!"); })
        .catch(function() { console.log("Share failure!"); });
    } else {
      // No web share API...
    }
  });
});

As you can see, most of the code here is setting up listeners, feature checking and getting the actual data to share. Breaking it down, here's what happens:

Select all links with the class "share"
Add a "click" listener to each link
When a share link is clicked, check for the existence of navigator.share
If it does exist, stop the link from doing anything
Try to get the canonical URL for the page, first from the link element if it is present, otherwise from window.location.href
Get the page title from the page's h1 element
Share the title and url

You can see the full code (at the time of writing) in GitHub.

Thoughts on the API

I like the API and I like the benefits it brings to both developers and users.

First up it's simple to implement. One method, 3 potential arguments, Promises for success and failure. You can dig a bit deeper into it, but on the surface a developer can have this implemented in a few minutes. Hopefully this simplicity will lead to swift adoption if the API becomes a standard.

There's more reasons I'm a fan though.

User choice

It stops host websites choosing where users can share content to. Sure, Facebook and Twitter are popular share destinations, but there are so many more apps out there that you could share content to. This long tail effect means that it is neither satisfying to only supply one or two share buttons, nor remotely sensible to make sharing everywhere possible. ShareThis may have a bunch of share buttons available, but the selection still doesn't scratch the surface of possibility across the web.

Ultimately, a share API like this puts the control back into the users' hands, allowing them to share to the applications they choose to install on their device.

Minimising JavaScript

You've seen the code above that I've used to implement the web share API. There's not much of it.

It's particularly small when you compare it to the scripts that popular sites supply to enable their share functionality. Twitter's script is ~32kB and Facebook's ~60kB (gzipped sizes). Best practice is to load those scripts asynchronously, so they shouldn't affect page load time. But that is a significant amount of JavaScript that needs to be loaded and parsed simply for a share button.

These scripts don't just function as the share buttons you are implementing as a developer. They allow Facebook, Twitter and the like to track their users around the web. The web share API not only gives site owners a more efficient way for users to share, but also gives back some user privacy.

Better feedback

While it may make it harder for large social networks to track users, the web share API actually makes it easier for sites to learn about their users' behaviour. As the share API works with Promises, you can find out when the share link is clicked and when a click results in a share to another application.

What's missing?

Currently the thing that is missing from the implementation of the web share API is the other side of it. The web share target API is an API that will allow web applications to register to appear in the share drawer when the web share API is invoked. Without the target API the web share API can only share directly to native applications available on a user's device. While this is probably still ok for those large social networks which already have applications installed on users' devices, this doesn't help the long tail of potential share targets.

I understand that becoming a share target is a much more complicated process than sharing. It is a good thing that the investigation of the share API and the share target API has been split up so that we can experiment with the web share API sooner. I hope to see the web share target API trialling ideas soon though.

Share away

The web share API is currently an experiment (which you can sign up to trial yourself) but I think it's an interesting and well meaning one. Ultimately giving users the choice over what they can share to is better for users and better for the long tail of applications.

What do you think of the web share API? Let me know on Twitter at @philnash. And if you're feeling generous, why not <a href="https://twitter.com/intent/tweet?text=The+web+share+API&url=https://philna.sh%2Fblog%2F2017%2F03%2F14%2Fthe-web-share-api%2F&via=philnash" class="share">share this post</a> too!

Doing things wrong

Mon, 13 Feb 2017 00:00:00 GMT

I wrote a blog post last week and bits of it were wrong. This is not a retraction of that blog post, I just wanted to share the feedback that I got, the things I changed and some lessons that I learned.

Inconsistency sucks

The first feedback I received was from posting the article on /r/javascript. The comment is deleted now so I can't credit the author, but they pointed out a few things:

Code styles between my two examples were different
Two semicolons were missing
It was better to format chains of Promises by placing the .then on the following line and indenting

I knew that the style of the two snippets was different as one had been written a year ago for the 12 Devs of Christmas blog and I had opted to retain the original code. But, mixing code styles in a project is unforgivable so why should a blog post be any different?

The formatting issues were also obvious once they had been pointed out to me. So, I went back to the post, improved the Promise formatting and fixed both the semicolons and the code style.

Over promising under delivers

The second piece of feedback came over Twitter, firstly in a direct message from Masse Fierro and then in a mention from James DiGioia. I'd gotten paranoid with my Promises and written one or two too many. I just needed to return the final data and not resolve another Promise.

   const punkAPIPromise = fetch(punkAPIUrl)
     .then(res => res.json())
-    .then(data => Promise.resolve(data[0]));
+    .then(data => data[0]);

Thanks to Masse and James I fixed that too.

Accessibility matters

The last thing to be pointed out, and probably the most important, was not about the content of the blog post but the accessibility of my site in general. James Spencer pointed out that my site failed colour contrast tests for both the code samples and links on the site. The tests they referred to are the Web Content Accessibility Guidelines on colour contrast which recommend a contrast ratio of at least 4.5:1 for regular body text. I learned some interesting things trying to fix this:

tota11y, by the Khan Academy, is a really awesome tool
- I learned about it from the link James shared with me
- It suggests colours that are similar to the existing colour that would pass the test
- It also has other plugins that visualise things like out of order heading usage, poor link text, missing alt attributes and a few more, helping you to ensure your site is as accessible as possible
Syntax highlighting themes are really bad at accessible contrast levels
- I was using a version of Solarized Dark which failed in a number of places
- I couldn't find any syntax highlighting theme that had decent contrast for all elements out of the box
- Using syntax highlighting in your editor is one thing, you can use whichever contrast levels you are comfortable with, but on the web anyone could be reading it

Eventually I chose to use the suggestions from tota11y to fix the four colours in Solarized Dark that were too low contrast. I also fixed my link text colour.

I was a little embarrassed about this because, as I said in a tweet, I thought I used to be good at this sort of thing. So much so that almost 10 years ago I wrote a blog post on the very matter of colour in web accessibility. This just goes to show that not only do we learn new things every day in web development, retaining old things is just as important.

Always be improving

I want to thank Masse, the two Jameses and the now mystery Redditor for giving me the feedback that helped to improve both my blog post and my site. If you see anything on this site that could be improved I welcome the feedback. I learned that I should keep my eye on the formatting and consistency of my code, not to over promise and to pay closer attention to accessibility concerns.

Overall I learned that in order to always be improving—my code, my site, myself—sometimes I need to do things wrong.

A toast to ES2015 destructuring

Thu, 09 Feb 2017 00:00:00 GMT

I think the ES2015 destructuring syntax is pretty cool. It might not grab the headlines like Promises, Generators or the class syntax, but I find it really useful. It's also surprisingly detailed when you read into it.

The only shame is that most of the examples I've found focus on just the syntax and not real life usage. Like this one from the MDN docs:

var a, b;
[a, b] = [10, 20];
console.log(a); // 10
console.log(b); // 20

Luckily I have a real life use case for destructuring for you right here.

Keeping Promises

One of my favourite uses for destructuring is when using Promise.all to wait for a number of asynchronous tasks to complete. The result of Promise.all is returned as an array and you would normally iterate over it or pick out the results you need by hand.

But, if you know what objects you are expecting to receive you can use parameter destructuring to make your life easier and your code both prettier and more descriptive by naming the parameters up front. Let's see an example.

Comparing beers

Let's say you want to compare two of BrewDog's delicious beers, something I find myself doing in real life all the time. We can get the information about them from Sam Mason's gloriously named Punk API. To implement this we use the fetch API to get the data on each beer from the API. We need both requests to resolve before we can compare the beers.

Let's take a look at the code:

  const punkAPIUrl = "https://api.punkapi.com/v2/beers/106";
  const deadPonyClubUrl = "https://api.punkapi.com/v2/beers/91";
  const punkAPIPromise = fetch(punkAPIUrl)
    .then(res => res.json())
    .then(data => data[0]);
  const deadPonyClubPromise = fetch(deadPonyClubUrl)
    .then(res => res.json())
    .then(data => data[0]);

  Promise.all([punkAPIPromise, deadPonyClubPromise])
    .then(beers => {
      const punkIPA = beers[0];
      const deadPonyClub = beers[1];
      const stronger = (punkIPA.abv < deadPonyClub.abv ? deadPonyClub.name : punkIPA.name) + " is stronger";
      console.log(stronger);
    });

We can tidy that Promise up with parameter destructuring:

  Promise.all([punkAPIPromise, deadPonyClubPromise])
    .then(([punkIPA, deadPonyClub]) => {
      const stronger = (punkIPA.abv < deadPonyClub.abv ? deadPonyClub.name : punkIPA.name) + " is stronger";
      console.log(stronger);
    });

We know we're getting two beers as a result and the way this example is constructed we even know which beer is which. So we can use the parameter destructuring to name the beers in the array instead of plucking them out.

More examples

Ok, this may still seem like a made up example, but it certainly is closer to real life. The first time I found myself using this technique was when writing about Service Workers for 12 Devs of Christmas. It came in handy when writing the method returnFromCacheOrFetch which implements the "stale while revalidate" caching method using the caches and fetch APIs.

The method opens up a named cache and tries to match the current request against the cache. Before returning, it then kicks off a fetch request for the requested resource, caching the result. Finally if the request was found in the cache the cached response is returned, otherwise the new fetch request is returned. You can read more about it in the original blog post.

The final code looked like this:

function returnFromCacheOrFetch(request, cacheName) {
  const cachePromise = caches.open(cacheName);
  const matchPromise = cachePromise
    .then((cache) => cache.match(request));

  // Use the result of both the above Promises to return the Response. Promise.all returns an array, but we destructure that in the callback.
  return Promise.all([cachePromise, matchPromise])
    .then(([cache, cacheResponse]) => {
      // Kick off the update request
      const fetchPromise = fetch(request)
        .then((fetchResponse) => {
          // Cache the updated file and then return the response
          cache.put(request, fetchResponse.clone());
          return fetchResponse;
        });
      // return the cached response if we have it, otherwise the result of the fetch.
      return cacheResponse || fetchPromise;
    });
}

In this case I needed both the result of caches.open and cache.match(request) in order to perform the background fetch and return the cached response. I drew them together using Promise.all and destructured the resulting array keeping the code tidy and descriptive.

Naming things

In these examples, parameter destructuring allows us to name the results we expect to get from the resolved Promises we pass to Promise.all. In fact, anywhere we use parameter destructuring, particularly with arrays, it allows us to name objects early and more descriptively. This in turn makes the code more readable and more maintainable in the long term.

Are there other places in your code that you've found the ES2015 destructuring syntax helpful? I'd love to know how you use the feature too, so please share your destructuring tips with me on Twitter.

On fixing a favicon

Fri, 27 Jan 2017 00:00:00 GMT

Sometimes open source work is just fixing one tiny thing that bugs you. However, rolling up your sleeves and delving into even the smallest amount of code can lead to surprising results.

I fixed a favicon recently. Here's what happened.

I fixed something for me

At the very least, fixing this issue has made me happier. The problem was that the Crystal standard library documentation wasn't showing a favicon, so when I had a bunch of tabs open in my browser I couldn't tell which ones had the Crystal docs I was using.

I fixed something for someone else

This is a guess, but if I found this an issue then we can presume at least one other person out there on the internet did.

I learned more about favicons

This was the bit I didn't expect. I've built websites for a long time now, I thought I knew all there was to know about favicons. How wrong I was.

I started by making a mistake. My original pull request was against the Crystal project itself and added <link> elements to the <head> of the documentation's HTML layout. This would make it dependent on the structure of the Crystal website though.

Feedback lead me to try again. This time I wanted to make the favicon for the Crystal website work by default for any pages situated on the crystal-lang.org domain, including the docs. This lead to a bunch of discoveries about favicons.

If you have a <link> for both a .png and a .ico favicon, Chrome and Safari will always pick the .ico
Firefox will always pick the .png
Internet Explorer only started supporting .png favicons since version 11

So, if you want to use a high quality .png favicon and a fallback for old Internet Explorer then the best course of action is to include a <link> to the .png icon in the <head> and place the .ico version in the default location: /favicon.ico. This way Chrome, Safari, Firefox and IE11/Edge will all choose the .png icon and old Internet Explorer will ignore the <link> and look for /favicon.ico.

Of course, as with anything on the web, there is even more to it than that. However, this was my resulting pull request and it was merged. Now favicons work on the documentation as well as the main site.

I learned more about the Crystal project

In my early pull request I learned about how the Crystal standard library documentation is generated and rendered. Then, when exploring the Crystal website I found that not only is it currently built on Jekyll but that there was an open issue pointing to my blog post on using Jekyll Assets to build an asset pipeline. If there was one thing in the world that I was definitely capable of it was using my own blog post to make caching assets better. So I rolled up my sleeves and prepared another pull request.

Now I am more familiar with the Crystal website and the team behind it that have commented on, reviewed and merged my pull requests.

The simplest piece of work can lead you anywhere

I just wanted to make favicons work yet it lead to 3 pull requests, a rabbit hole of discovery about favicons and browsers and the motivation to write a blog post about it. Ultimately, I did get what I wanted in the first place though: more recognisable tabs when I'm working with Crystal.

Open source is a lot of things to a lot of people. To me, it's a way to learn while I help, however small the task may seem. I'd love for you to share with me on Twitter what open source means to you.

Dev Tools Tricks: Store objects and elements as variables in the console

Thu, 12 Jan 2017 00:00:00 GMT

Browser dev tools are so full of features it's hard to keep up. I bet every developer knows a different set of features to each other. I wanted to share a few little tips that I use, that you might not know and that if I don't write down I might forget.

Read on and discover the magic of $0, $_ and temp0.

Access selected elements as variables

When you're clicking around the Inspector in Firefox or the Elements tab in Chrome and you want to use the selected element in the console, you need a reference to that element. You could type out a querySelector that would pick the element out of the document, but that's far too much work. Instead, leave the element selected and use $0 in the console to get the reference you need.

Chrome even hints at this by showing == $0 next to the HTML.

Be aware though, if you select a different element in the inspector then it will become the subject of $0 and you will lose the reference to the old one.

Firefox has another trick for this. Right click on an element and choose "Use in console". This places a variable called temp0 into the console's command line which is a reference to that selected element.

Turn any object in the console into a global variable

If you're dealing with other objects in the console, perhaps something you've logged from your own code, and you want to reference them then there are a couple of ways to do that too.

$_, for example, is a reference to the last object that was returned in the console.

More generally, any object that has been returned or logged to the console can be turned into a global variable by right clicking on it and selecting "Store as global variable". You will get a variable called temp0 which references that object. Even better, do it for more objects and you'll get new variables called temp1, then temp2 and so on.

Dev tools go deep

There is so much to learn about dev tools, these are just a few little tricks that might help when debugging your code. If you're looking for more tips like this, I recommend signing up to Umar Hansa's Dev Tips or watch his ffconf 2016 talk.

Don't forget to share your own tips or things you've found in dev tool. If you've got any good ones, please send them to me on Twitter at @philnash.

Git back to the future

Wed, 04 Jan 2017 00:00:00 GMT

Git may be the best version control software I've used but it is a complex beast and makes it easy to shoot yourself in the foot. Recently, however, I learned of one way that you can unshoot yourself and potentially save yourself hours of lost work.

git reset is useful when you've done something wrong. You can turn back the clock and undo commits.

git reset --soft HEAD~1 undoes one commit and leaves the work from that commit still present in the working directory.

git reset --hard HEAD~1 removes the commit and all the work.

The difference between the --soft and --hard flag is one of those ways you can shoot yourself in the foot. I know I've mixed the two up and lost work. Or so I thought.

Enter the reflog

It turns out that git is watching us closer than we may think. When we make changes to any branch, git stores those changes in the reflog. Even if we were to remove a whole bunch of work by resetting with the --hard flag, git still knows about the commits we had made and we can recover them. The key is that the commits still exist, there just aren't any branches that currently point to them. This is where the reflog comes into play. It has a log of all commits made in the repo, as well as other actions, and we can use it to recover these lost commits.

Warning!

Before I show you how this works, please note the following. If you have uncommitted changes in the working directory and you use git reset --hard no amount of fancy git knowledge about the reflog is going to get that back. The following will save only work that you have committed. Be warned!

How to use the reflog to recover lost commits

Here's an example, I have a repo with one commit. Running git log --oneline and git reflog both show the commit has the hash 2daf3ba.

Now we make a commit, something important of course.

What happens if we git reset --hard HEAD~1?

git log shows only one commit, but git reflog shows three actions; two commits and one reset. Note how the reset points to the same hash as the original commit. So, how do we get back to the last state before we reset? We can reset again.

This time we reset using the hash of our lost commit. You can always reset using hashes, in fact HEAD~1 is really just a reference to a hash. What does the reflog look like now?

The reflog now shows the four actions that we took; two commits and then two resets. Now we have reset the branch to the state we were in before, no work has been lost, breathe a sigh of relief.

May you never lose your work again

The git reflog keeps a track of everything we get up to with git. As with most things in git once you learn about the feature you can use it to your advantage. Knowing I have this trick up my sleeve has made me more confident using git.

I'd like to thank Steve Smith for the talk he gave at Codemotion Berlin where I learned this trick. There is an audio recording of his talk available and a much more detailed article about git refs and the reflog on Atlassian's tutorial site.

If you need more tips for escaping from nightmare git scenarios, check out Oh shit, git! a collection of git tips from Katie Sylor-Miller. She calls the reflog a magic time machine, I certainly agree and hopefully now you do too.

<footer> <small>Git logo by <a href="https://twitter.com/jasonlong">Jason Long</a>.</small> </footer>

A community offers to help

Mon, 05 Sep 2016 00:00:00 GMT

Last week I was blown away by the response to a problem I was experiencing from maintainers of two of the larger open source projects in the Ruby world. And not just your average large Ruby projects, but implementations of Ruby itself.

I had trouble running the tests for envyable, my gem to manage environment variables in your projects, on Rubinius on TravisCI. After installing the version of Rubinius locally, I couldn't reproduce the issue. So I tweeted.

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Anyone seen this error on <a href="https://twitter.com/travisci">@travisci</a> with <a href="https://twitter.com/codeclimate">@codeclimate</a> coverage on <a href="https://twitter.com/rubinius">@rubinius</a>? <a href="https://t.co/fydiHoElCa">https://t.co/fydiHoElCa</a>. I can't repro locally and am confused.</p>— Phil Nash (@philnash) <a href="https://twitter.com/philnash/status/770542433901969409">August 30, 2016</a></blockquote>

Not only did Brian Shirai, maintainer of Rubinius, reach out to suggest some ideas for how to fix it. He also forked the project and tried to get it working himself.

<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/philnash">@philnash</a> you're using an ancient version 3.29. Ensure you're using Trusty distro and 'rbx' in .travis.yml <a href="https://twitter.com/travisci">@travisci</a> <a href="https://twitter.com/codeclimate">@codeclimate</a></p>— Rubinius (@rubinius) <a href="https://twitter.com/rubinius/status/770647800514093056">August 30, 2016</a></blockquote>

But that wasn't all. I followed his advice but eventually had to settle with moving Rubinius to the allowed failures list in my TravisCI config. I'm still working with the TravisCI support team to try to fix the issue. We're getting there as it turns out TravisCI was looking in the wrong place for the latest Rubinius binaries. That issue is fixed but I am now getting timeouts for my builds.

This affected JRuby too and when I mentioned that on Twitter, Charles Nutter from the JRuby core team also offered to help out if it was a JRuby bug.

<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/philnash">@philnash</a> <a href="https://twitter.com/travisci">@travisci</a> <a href="https://twitter.com/rubinius">@rubinius</a> That JRuby failure looks like a timeout installing the base gemset in rvm. Happy to help if it is a JRuby bug.</p>— Charles Nutter (@headius) <a href="https://twitter.com/headius/status/770792465955557376">August 31, 2016</a></blockquote>

My project is not the biggest or the most important Ruby project. Not by a long way. The issues are also likely not with Rubinius or JRuby. But the willingness for Brian and Charles to reach out to help was amazing. I'm sure they are very busy people and just the response was appreciated.

Thank you

So thank you Brian and thank you Charles. Thank you for your work in the Ruby community and thank you specifically for offering your help to me. Acts like this maintain my belief in the power and the strength of the Ruby community.

Install a service worker declaratively

Wed, 17 Aug 2016 00:00:00 GMT

There's been some interesting updates in service workers recently. The big news is that the Microsoft Edge development version now has service workers, alongside push notifications and background sync, behind flags. There was a new feature that caught my eye though; a declarative method for registering service workers.

Up until now, to register a service worker, you need to write JavaScript. It would look something like this (emoji logging optional):

<script>
  if ('serviceWorker' in navigator) {
    navigator.serviceWorker.register("/sw.js", { scope: "/blog/" })
      .then(function(registration) { console.log("🍻"); })
      .catch(function(error) { console.log("😭", error); });
  }
</script>

If you allow the service worker lifecycle to play out naturally and you're not using push notifications or background sync then this is all you need. New in Chrome 54 (currently Canary) there is an experiment to make this even simpler, more declarative, with the use of a <link> tag in the HTML or via a Link header.

I've put together a quick example repository showing all three approaches which you can use to install a service worker. There's the JavaScript way, as seen above. Or, you can just use a <link> tag, like so:

<link rel="serviceworker" href="/sw.js" scope="/blog/">

Or, you can install the service worker with an HTTP header that looks like this:

Link: </sw.js>; rel="serviceworker"; scope="/blog/"

Each of these methods will install the service worker file /sw.js with the scope "/blog/". I've produced an example of how you'd use these methods to install a service worker on GitHub. Make sure to enable "Experimental Web Platform features" in Chrome Canary before you try it out.

Foreign fetch

It turns out that these alternative ways to register a service worker stem from the foreign fetch proposal.

When you have a service worker registered it can respond to any fetch request made from a page under its control. Foreign fetch intends to make it possible to register a service worker and respond to cross origin requests. You can read more about the foreign fetch proposal in the spec on GitHub. The take away for me was that foreign fetch will make it more robust to rely on third party APIs that you call from the browser in our web apps.

As you can imagine, cross origin requests for resources like an API won't be running your regular JavaScript that would install your service worker. This is where the Link header comes in.

The service worker revolution marches on

Whether it's new developments in browsers that already support service workers or new browsers joining the fold (come on Safari…), I continue to be excited about the potential this technology is bringing to our web applications. I particularly like simplifications to the process and installing a service worker with just a <link> tag appeals to me.

If you want to catch me speaking about my excitement for service workers I'll be doing so all over Europe this Autumn. Look out for me at Frontend Conference Zurich, Refresh in Tallinn and DevDay in Kraków. I hope to see you there!

August is iOS month

Mon, 01 Aug 2016 00:00:00 GMT

June saw me ask for your help as I started to learn Swift. July was busy with travel. Now August is iOS month.

I'm lucky that part of my job as a developer evangelist for Twilio is to learn new things. That is why I attended the Big Nerd Ranch Swift and iOS course in June and is what has lead me to dedicate August to building my first iOS application.

My first app

By first application, I really mean my first one that I am going to take from beginning to end, from blank Xcode project to the App Store, all by myself. I have worked on iOS applications before. I remember a time when I managed to crash an iPad app so badly while working on it, that checking out the previous version of the app still lead to crashes in the feature that I'd spent too long looking at. I did not receive too many more tasks to complete on that application. But those were the days of memory managed Objective C and working in a team.

Now, with my newly gained knowledge of Swift, it's just me, Xcode and a month of clear space to build something.

2,000,000 applications

Of course, the question one always lands on is what that something should be. This is my first application and a solo effort at that, so I'm looking for something with a constrained scope, something that will be achievable this month, something that will be achievable by me. When I remembered that Apple announced at WWDC this year that the App Store now had more than 2 million applications available to download I realised that I'm not looking for a brand new idea. Anything that can be built by a novice developer within a month has almost certainly already been built.

I decided the best idea would be to tackle something I have done before in a different environment. As it happens, I own fxrat.es, the world's least popular exchange rate calculator. It's best feature is not up to the minute exchange rate data, but that it works offline in modern browsers. It's a web application I wanted to use myself when abroad.

New and improved

I have been thinking about rewriting it for a couple of reasons. Firstly, sometimes it just doesn't work and I've never found out why. Secondly, it currently works offline using the web platform's Application Cache. The AppCache is going away, for good reasons, so it is time to update to use my new favourite browser feature, the Service Worker. However that will not work in Safari.

This seems to me like a good opportunity to both rewrite my existing application for the web and produce a new, offline capable native application for iOS that I can release in the App Store.

The task is set

So that is August for me. Buried in Xcode creating a new application. Wish me luck and follow my progress on GitHub.

See you in the App Store.

Creating an asset pipeline with Jekyll-Assets

Tue, 28 Jun 2016 00:00:00 GMT

When I started again on this site, I wanted to make sure it was going to load fast. Performance matters.

Bundling, minifying and caching the static assets was high on the priority list. The site is built on Jekyll so I went looking for (and ultimately found) a plugin that would perform the following:

Bundle static files together
Minify the resulting files
Generate a hash of the content of the file as the production filename (the Rails documentation has a good explanation of this fingerprinting technique)

Can't Jekyll do this?

Jekyll 3 can handle the bundling and minification of CSS. In fact, Jekyll comes with Sass included which actually does the heavy lifting.

When you start a new Jekyll site, you will find a css directory with a main.css file present. That file then uses Sass imports to require other files from the _sass directory.

That's bundling handled already, how about minification? It takes one config update to enable that too. Add the following to your _config.yml file:

sass:
  style: compressed

Jekyll-Assets for all the features

If you're hosting a Jekyll site on GitHub Pages this is as good as it gets. You can't install other Jekyll plugins and GitHub handles the caching headers for your static files.

But what if you have your own server and want more control? What if you want those fingerprints so that you can set your own cache headers far into the future? What if you want to provide your own transpilers, processors or minifiers?

Those are the features I wanted. So I did a bit of research and found Jekyll-Assets. It's a pretty powerful plugin, with lots of features for optimising assets. Here's how you can get started using it.

Getting started with Jekyll-Assets

<div class="info"> <p>This blog post was written for Jekyll Assets version 2.4.0. Jekyll Assets version 3 has now been released and changes a few things, if you are using Jekyll Assets 3 the following may not apply.</p> </div>

Let's set up Jekyll-Assets for a Jekyll site. First up, we need to install the jekyll-assets gem. Add it to the Gemfile:

gem "jekyll-assets"

and install with bundle install. Then add it to the _config.yml file.

gems:
  - jekyll-paginate
  - jekyll-assets

Jekyll-Assets comes with a comprehensive default config to make it quick to get started, so we don't need to change that right now. We do need to move a couple of files about though.

As we saw earlier, Jekyll comes with a css directory containing a main.css file and then a _sass directory with a few Sass imports. We want to move them all to the _assets/css directory.

$ mkdir -p _assets/css
$ mv css/main.css _assets/css
$ mv _sass/* _assets/css

Jekyll includes some empty front matter in main.css. We need to get rid of that too.

---
# Only the main Sass file needs front matter (the dashes are enough)
---
@charset "utf-8";

It may say that the main Sass file needs front matter but with Jekyll-Assets that is no longer the case.

Fixing up the HTML

The CSS is now no longer where the layout thinks it should be. We need to update the _includes/head.html to point to our new CSS file. As we are going to be generating unique filenames for our assets in production we need to use the Jekyll-Assets supplied liquid tags to fill them in.

Open up _includes/head.html and replace the existing <link> with:

{{ "{% css main " }}%}

If you want to see the filename digests working a quick config change is all you need. Open up _config.yml and add:

assets:
  digest: true

Start the Jekyll development server with bundle exec jekyll serve and open up http://localhost:4000. The site will be displaying as you expect and if you view source you will see the generated <link> tag looking a little like this:

<link type="text/css" rel="stylesheet" href="/assets/main-daf4744048a36abfd0aae160e2f7c309c4ae468f16d65e77d89c70fc8d7ba6ec.css">

Our digests are working! You can turn this off in development mode, Jekyll-Assets will generate digests, as well as bundle and minify, by default in production mode.

Other assets

If you want other assets to get the Jekyll-Assets treatment, all you need to do is move them to their respective directory in _assets and refer to them using the relevant liquid tags. The default available directories include _assets/images and _assets/js.

On the server

The entire purpose of adding Jekyll-Assets to my site was to set cache headers way into the future so that browsers can cache the assets as aggressively as they can. The easiest part of this turned out to be adding the config to my web server. I'm using nginx and all I needed was:

server {
  # other config

  location /assets/ {
    expires 1y;
    add_header Cache-Control "public";
  }

  # ...
}

Bundled, minified, digested, cached

Using the Jekyll-Assets gem and a bit of nginx config I optimised the static assets on this site. There are loads more options for the gem including integrations with Babel for ES2015, CSS Autoprefixer and Image Magick for image optimisation. It is worth exploring the code and the documentation to see what else can be achieved.

Now my assets are nicely optimised it's more fun to run WebPageTest against the site. Looks like I better start looking into CDN options now. Thankfully Jekyll-Assets supports those too.

*[CDN]: Content Delivery Network *[CSS]: Cascading Style Sheets *[ES2015]: ECMAScript 2015 *[HTML]: Hypertext Markup Language

Critique my Swift on Exercism

Mon, 20 Jun 2016 00:00:00 GMT

I am currently in the middle of a Swift and iOS course with the excellent Big Nerd Ranch. Taking a week to dedicate to learning a new language and framework means I can block out all other distractions and really focus. And as this goes on, I'd like your help.

Specifically, I'd like you to log on to Exercism and critique my code.

Practice makes…?

Exercism is a collaborative tool for learning, practicing and reviewing code created by the fantastic Katrina Owen. It's great for beginners, to a new language or to coding in general, to try out new skills and get reviews and tips from peers. It's also great for experienced programmers to keep sharp by writing and reading code. And it's all open source and has a great page on how to contribute, which is awesome.

I'm currently trying to use it to kickstart my Swift career. That's where you come in.

Over the coming days and weeks I will be completing the Swift problems set out by Exercism and I'd really like a second (or third, or fourth) opinion. If you know Swift ,or if you don't, all opinions are welcome, I would love it if you would check out my attempts and leave some pointers on how I might improve.

Hello, World!

Just to kick it off, here's my "Hello, World!" from the first Exercism challenge in Swift.

class HelloWorld {
    static func hello(name: String = "World") -> String {
        return "Hello, \(name)!"
    }
}

It may only be five lines of code, but it's a start!

Improving together

This works better if we learn and feedback to each other, so I'd love to help you too! If you're using Exercism to practice or learn something new, I am available to send feedback. If you want me to take a look at your code just give me a nudge on Twitter.

Exercism is a great tool and an opportunity for us all to improve together. So what are you waiting for?

The surprise multipart/form-data

Mon, 13 Jun 2016 00:00:00 GMT

Building up and sending an Ajax request is so much easier than it ever used to be. No longer must we hassle ourselves with XMLHttpRequest, never mind the horror of ActiveX's ActiveXObject("Microsoft.XMLHTTP"). The Fetch API is here (and it is polyfilled for older browsers). Then there's the FormData object that makes building up and submitting form data really easy, especially compared to the 130 or so lines of JavaScript you'd need to do it yourself.

But you can still get things wrong. I can definitely get things wrong.

So I wanted to write about what I got wrong to remind myself for the future. My mistakes involved the Fetch API, FormData and a simple Express server in Node.js.

I couldn't POST any data

It was simple, I wanted to POST some data to my server. I had Express set up, I'd installed body-parser (normally something I forget to do until this kind of problem comes along), I had a route that was ready to receive my data and do something with it.

It looked a little like this:

const Express = require("express");
const bodyParser = require("body-parser");

const app = new Express();

app.use(bodyParser.urlencoded({ extended: true }));

app.post("/", function(request, response) {
  console.log(request.body.foo);
  response.send("OK");
});

app.listen(3000, function() {
  console.log("Application is listening on localhost:3000");
});

OK, it was more complicated than that, but that's all we need to demonstrate this.

The front end was going to be even easier.

var formData = new FormData();
formData.append("foo", "bar");
fetch("/", {
  method: "POST",
  body: formData
}).then(function(response) {
  // Yay!
}).catch(function(err) {
  // Boo!
});

The plan was simple: run the server, run the front end code, see "bar" printed in the console, celebrate.

undefined, the JavaScript developer's worst enemy (aside from perhaps callback hell and developers from other languages telling them that they're doing it wrong).

I searched up and down the request object, looking for that simple piece of data that I had sent. I checked in all the places I thought it might be request.body, request.params, request.query, Express has all of those you know. I checked in plenty of places I didn't think it would be. Finally I did what any sensible developer would do. I asked the internet.

The internet is a wonderful support network

I love that out there is a resource where I can ask questions and people respond and help me. The internet came up trumps in this case, I asked and many people chimed in with ways to fix the issue.

The first solution was to set the content type to JSON and just send JSON to the server.

var data = { foo: "bar" };
fetch("/", {
  method: "POST",
  body: JSON.stringify(data),
  headers: {
    "Content-Type": "application/json"
  }
}).then(function(response) {
  // Yay!
}).catch(function(err) {
  // Boo!
});

This worked, but I was confused. I had this really easy FormData object to work with, but the solution was to throw it away and use JSON.stringify? It didn't seem right. It worked but I wanted to persist and make it work the way I had intended.

It works in Rails

Eventually Edd sorted me out. Firstly he said that my front end JavaScript wasn't wrong, the same code worked with a Rails back end. I think that put him on the route to find the correct answer though.

It turns out that body-parser doesn't understand multipart/form-data encoded forms (though as Edd pointed out, Rails does). And FormData, when added to a Fetch API request (or an XMLHttpRequest as it happens, I checked), will set the content type to multipart/form-data. It actually says that right here in the first paragraph of the MDN documentation. In bold.

I like to say that "people don't read things on the internet". I don't read things on the internet.

In my defence, I would have thought that body-parser might parse the body of the request. Once again skimming over the documentation was my downfall. Check out line 2 of the body-parser readme.

In fact, the body-parser readme doesn't just say that it won't handle multipart/form-data bodies, it even goes as far as suggesting other modules to use for those requests.

I ended up using formidable via the express-formidable middleware. Our example server now looks like this:

const Express = require("express");
const bodyParser = require("body-parser");
const formidable = require("express-formidable");

const app = new Express();

app.use(bodyParser.urlencoded({ extended: true }));
app.use(formidable());

app.post("/", function(request, response) {
  console.log(request.fields.foo)
  response.send("OK");
});

app.listen(3000, function() {
  console.log("Application is listening on localhost:3000");
});

Lessons learned

FormData objects set the content type of a Fetch API (or XMLHttpRequest) request to multipart/form-data automagically.

body-parser doesn't support parsing the body of a multipart/form-data request. But there are many options that do.

The internet is a very helpful place. Thank you to Edd and others who suggested what I might be doing wrong.

READ THE DOCUMENTATION. Then probably read it again just to make sure.

There are no surprises

There are actually very few surprises in development, particularly when using modern browser APIs and well supported frameworks and reading the documentation. I learned quite a bit from this one small episode as you can see above. The next thing I need to do is write some documentation for someone else. Eventually, they'll read it and be thankful as I was.

This site isn't finished

Wed, 18 May 2016 00:00:00 GMT

No running site is ever completely finished. This one is no exception.

There is more to build, more to style and more to write. It's a challenge.

Welcome to the new version of my site. The old one was notable for it's long service and disappointing content. It's better left forgotten.

This site is finished.

So, here's to a fresh site, new writing and a place to experiment with the web. I'm quite looking forward to it.