Here is a LogRocket article explaining Protobuf in depth

All of these technologies use something called a “schema”. This is used to serialize payloads into a binary and deserialize the binary into a payload. They offer type-safety in that the processes will fail if the payloads are not of the correct schema structure. Typical is based on functional principles, is relatively new to the scene with being around for two years and is maintained by a small set of core developers. In comparison, Protobuf has fifteen years behind it and is maintained by Google. Both have a lot of similarities with the main differences accounting for the more modern features Typical has to offer. Typical has unique features like “asymmetric” fields for ensuring backward/forward compatibility and “choices” for supporting pattern-matching, while Protobuf support plugins and has a long-standing reputation and widespread use make it a more battle-tested solution for many.

To showcase the two technologies, we will go through some common tasks a developer will encounter when working with schemas when working with TypeScript. TypeScript is a very popular programming language which is a superset of JavaScript. We will go through a detailed tutorial working with employee data on how to serialize/deserialize data, how to safely make schema changes and what options there are for optional fields.

Our Example Employee service and Schemas

Lets pretend we have a service which sends data about employees to other services within an organization, lets call this service the “writer” (producer, serializer, publishers) when it comes to data interchanges and schemas. We will also handle receiving this employee data. These services can be called “readers” (consumers, deserializers, subscribers). We could use Kafka, GCP Pub Sub or a myriad of other technologies to send the data which supports binary transports. For the sake of examples and our tutorial, lets assume our services are written in TypeScript and are running on Node.js.

We will get our service setup to use both Typical and Protobuf so we can demonstrate the differences. The great thing about both of these technologies is that both offer a CLI tool to generate code based on a schema. It will generate serializers, deserializers and TypeScript types. This means we can use the schema as the source of truth for what payloads should look like, if changes need to be done, we can do it in the schema and run the generation over again. This can save you a lot of time and headache for developers who want to speed along coding.

All of the code for this blog post can be referenced to this github repository

Setting up our development machine

To set everything up, we can run this script. We will install the Typical and Protobuf runtimes via brew. Ill be explicit how which steps are needed for each. The runtimes are needed for various functions like generating the types and code, etc.

It would be nice if these came packaged up on NPM but I couldn’t find packages suitable. Let me know if you find a better way.

We will create a new folder for the project, create a package.json file with npm init and get TypeScript installed.

mkdir typical-vs-protobuf-example & cd typical-vs-protobuf-example
brew install typical
brew install protobuf
// -- fix mac os js protobuf compiler issue
brew install protobuf@3
brew link --overwrite protobuf@3
// --
npm init -Y
npx tsc --init
npm add -D typescript ts-node @types/node
// Needed for TypeScript type gen in protobuf
npm add -D ts-protoc-gen

The Schema

Here is a sample payload structure for the data we will be sending from service to service:

{
    "id": "1",
    "name": "John Doe",
        "hourly_rate": 20,
    "department": "HR",
    "email": "john@doe.com",
    "active": true
}

The next step will be to define a schema for the payload, this defines what the payload should look like, if it doesn’t follow the schema, serialization and deserialization when running the service will fail.

This is common functionality between most data interchange technologies. The schema is what provides “safety” to services and clients that they will receive the correct data in a shared format.

In both Typical and Protobuf, the schemas will look similar with minor differences. We will define an Employee struct which contains basic information like name and hourly_rate, we will use an enum to offer a set of choices for departmentinstead of a string as we only have two options it can be.

Typical:

// types.t

struct Employee {
    id: String = 1
    name: String = 2
    houry_rate: F64 = 3
    department: Department = 4
    email: String = 5
    active: Bool = 6
}

choice Department {
    HR = 0
    NOT_HR = 1
}

Protobuf:

// types.proto

syntax = "proto3";

package employee.v1;

message Employee {
    string id = 0;
    string name = 1;
    string email = 2;
    int32 hourly_rate = 3;
    Department department = 5;
    string email = 6;
    bool active = 7;

    enum Department {
        HR = 0;
        NOT_HR = 1;
    }
}

Once we have those defined, we can move onto the TypeScript code.

Generating Serializers/Deserializers

Once we have the schema defined we can generate the TypeScript types and code using some NPM scripts. Add these scripts to your package.json:

{
    "scripts" {
            "generate:typical:types:1": "typical generate typical-example/types-1.t --typescript typical-example/generated/types.ts",
        "generate:protobuf:types:1": "protoc --plugin=\"protoc-gen-ts=./node_modules/.bin/protoc-gen-ts\" --ts_opt=esModuleInterop=true --js_out=\"./protobuf-example/generated\" --ts_out=\"./protobuf-example/generated\" ./protobuf-example/types-1.proto"
        }
}

Run the generate:typical:types:1 cmd and inspect the generated code in typical-example/generated/types.ts. Do the same for same for Protobuf but the path is, protobuf-example/protobuf-example/typeos-1_pb.ts.

Now we will write some code which uses it to serialize our sample payload from above into a binary. First in Typical:

import { Types1 } from "./generated/types";

// Take our sample payload
const payload = {
  id: "1",
  name: "John Doe",
  hourlyRate: BigInt(20),
  department: { $field: "hr" as const },
  email: "john@doe.com",
  active: true,
};

// Serialize the Employee object to binary using the generated Serializer from Typical
const binary = Types1.Employee.serialize(payload);

// Log that it was successful
console.log("Successfully serialized Employee object to binary:", binary);

// Send the binary off using Kafka, etc
...

Then in Protobuf:

import { Employee } from "./protobuf-example/types-1_pb";

// Take our sample payload
const payload = {
  id: "1",
  name: "John Doe",
  hourlyRate: 20,
  department: Employee.Department.HR,
  email: "john@doe.com",
  active: true,
};

// Create the Employee object based on our sample payload
const employee = new Employee();
employee.setId(payload.id);
employee.setName(payload.name);
employee.setHourlyRate(payload.hourlyRate);
employee.setDepartment(payload.department);
employee.setEmail(payload.email);
employee.setActive(payload.active);

// Serialize the Employee object to binary
const binary = employee.serializeBinary();

// Log that it was successful
console.log("Successfully serialized Employee object to binary:", binary);

// Send the binary off using Kafka, etc
...

As you can see, there is still is not many differences. When it comes to schema changes and optionals is where things start to change.

Making a schema change or schema evolution

We realized we made a crucial error, we need to add an employee’s phone number to the schema. All employees have to input the phone number into the service so it will always be present. After some thought we decide to make this field required. Depending on the technology used and the quantity of readers and writers, it may be a breaking schema change. This means that we need to be careful to not send incompatible payloads to services which can’t handle them.

For example, let’s say we have multiple writers using this schema, if we add a required field, some writers will not be updated at the same time (we are part of a big organization). If this is the case readers can’t expect it to be present on every single message until every writer has there code and schema updated.

This is a big topic when it comes to messaging systems so we will only focus on the differences between Typical and Protobuf to go about this kind of change.

The question is how do we roll this change our safely and effectively. There are other changes we may want to make.

Some definitions:

• Backwards Compatible
  • Means writers send messages of a new schema version and readers can still process messages using an old schema version.
• Forwards Compatible
  • Means readers updated to a new schema version can still process messages with writers using an old schema version.

First lets dig into how Typical does it. Every change is forwards and backwards compatible which makes things easy to understand. There is a feature called “asymmetric” fields which were made for this use-case.

There are no non-nullable fields in Typical, which is a big difference compared to other technologies.

How this works if that you add the keyword on a field. This basically says that its required for the writer, optional for the reader. When we are 100% sure all of the writers have been updated, we remove the keyword which makes the field required. All fields in Typical are required by default.

Lets see an example schema:

struct Employee {
    id: String = 1
    ...
    asymmetric phone_number: String = 7
}

Now that we have the schema, run type generation again and inspect the outputted types:

export type EmployeeOut = {
  id: string;
  ...
  phoneNumber: string;
};

export type EmployeeIn = {
  id: string;
  ...
  phoneNumber?: string;
};

You can see how it is now optional for the readers. Here is a summary of schema changes which are safe in Typical.

Now with Protobuf, all fields are required by default. But the main difference is that there is a traditional optional keyword which you can use. This makes the rules on how you can evolve a schema a more involved. Some changes are forwards compatible and some are backwards, overall its more granular approach.

Here is a good article which summarizes what kind of changes are forwards/backwards compatible

Now our strategy must goes as follows:

1. Update the Protobuf schema to have phoneNumber as optional
2. Update all writers to use the new schema and update the code to have the value be present
3. Update all readers with the new schema
4. Wait until all writers have finished updating
5. Update the Protobuf schema to have phoneNumber as required
6. Repeat steps 2, 3, and 4

You can see it’s more involved and you have to know what you are doing. This process is similar to Thrift and Avro.

Flexible Payloads

You may have payloads where optional fields make sense. An example could be filling the success and error fields inside of a payload. When a success message is sent, error would not be present and vice verse.

In Typical, the spec doesn’t support non-nullable fields but instead uses something called “choices”. This offers pattern-matching-like capabilities to readers and acts like an enum (we used it before for department). It’s much more flexible than a traditional enum as fields inside of a choice can be strings or new structs.

Here is an example of adding a details field to the Employee struct which explains to a reader if it was successful or an error occurred in the payload.

Here is the Typical schema:

struct Employee {
    id: String = 1
    ...

    details: Details = 7
}

choice Details {
    success = 0
    error: String = 1
}

Here is what the writer may serialize:

const payload: Types2.EmployeeOut = {
  id: "1",
  ...
  details: {
    $field: "success",
  },
};

Here is what the reader may deserialize:

// Read the binary payload from a file
const fileContents = readFileSync(filePath);
// Deserialize using Typical generated code
const payloadDeserialized = Types2.Employee.deserialize(
  new DataView(
    fileContents.buffer,
    fileContents.byteOffset,
    fileContents.byteLength
  )
);
// Handle the details field
switch (payloadDeserialized.details.$field) {
  case "success":
    console.log("We have a success!");
    break;
  case "error":
    console.log("We have an error!");
    break;
  default:
    throw new Error("Unknown details field");
}

In Protobuf you can use the optional keyword like we described before. If you want to mimic the behaviour of a choice in Typical, there is a keyword oneof. This can codify different options a field can take and you can define more than a traditional enum.

Here is an example of using the same schema as above using a oneof :

message Employee {
    string id = 1;
    ...

    oneof details {
        bool success = 7;
        string error = 8;
    }
}

Here is what the reader may deserialize:

...
// Read the binary payload from a file
const fileContents = readFileSync(filePath);
const payloadDeserialized = Employee.deserializeBinary(fileContents);

// Handle the details field
switch (payloadDeserialized.getDetailsCase()) {
  case Employee.DetailsCase.SUCCESS:
    console.log("We have a success!");
    break;
  case Employee.DetailsCase.ERROR:
    console.log("We have an error!");
    break;
  default:
    throw new Error("Unknown details field");
}

Conclusion

While Typical offers some unique features like asymmetric fields for ensuring backward/forward compatibility and choices for pattern-matching, Protobuf's long-standing reputation and widespread also makes it an attractive choice. Being a relatively new technology, Typical can offer some advantages like a single cohesive CLI tool which can generate types and code instead of having to download separate packages in Protobuf. On the other hand, Protobuf supports plugins and has a wider range of language support compared to Typical. It also offers traditional optional fields, making schema evolution more involved but also more granular when compared to the opinionated approach of Typical.

In conclusion, both Typical and Protobuf have their own unique advantages and limitations. The choice between the two depends on the specific needs and preferences of the organization.

One example is the reintroduction of decorators in the soon-to-be-released TypeScript 5.0; decorators is a meta-programming technique that can be found in other programming languages. If you’re an application developer or library author who is interested in using the new official TypeScript decorators, you’ll want to adopt the new syntax and understand the differences between the old and new feature sets. The API differences are extensive and it is unlikely that old decorators will work with the new syntax out of the box.

In this article, we’ll review the history of using decorators in TypeScript, discuss the benefits associated with decorators in TypeScript 5.0, walk through a demo using modern decorators, and explore how to refactor existing decorators.

N.B., all the APIs have changed extensively in TypeScript 5.0; for this article, we’ll focus on class method decorators

Jump ahead:

• History of TypeScript Decorators

• Decorators in TypeScript 5.0

• Decorator factory demo

• Refactoring existing decorators

• Understanding the limitations of modern decorators

History of TypeScript Decorators

Decorators is a feature that enables developers to reduce boilerplate by quickly adding functionality to classes, class properties, and class methods. When TypeScript first introduced decorators it did not follow the ECMAScript specification. This wasn’t great for developers, since ideally emitted code from any JavaScript compiler should comply with web standards!

Using decorators required setting an --experimentalDecorators experimental compiler flag. Several popular TypeScript libraries, such as type-graphql and inversify, rely on this implementation.

Here’s an example of a simple class method decorator, demonstrating the enhanced ergonomics of the new syntax:

function debugMethod(
  _target: unknown,
  memberName: string,
  propertyDescriptor: PropertyDescriptor
) {
  return {
    get() {
      const wrapperFunction = (...arguments_: unknown[]) => {
        const now = new Date(Date.now());

        console.log("start time", now.toISOString());

        propertyDescriptor.value.apply(this, arguments_);

        const end = new Date(Date.now());

        console.log("end time", end.toISOString());
      };

      Object.defineProperty(this, memberName, {
        value: wrapperFunction,

        configurable: true,

        writable: true,
      });

      return wrapperFunction;
    },
  };
}

class ComplexClass {
  @debugMethod
  public complexMethod(a: number): void {
    console.log("DOING COMPLEX STUFF!");
  }
}

In the above code, we can see that the debugMethod decorator overrides the class method property using Object.defineProperty, but in general, the code isn’t easy to follow. Also, the arguments are not type-safe, which limits our safety inside the wrapperFunction. Additionally, the compiler will not fail if this decorator is used on an invalid use case, such as a class property.

We could use TypeScript generics to try to achieve type safety, but TypeScript does not infer generic types and this makes them a pain to consume. Thus, writing complex decorators is difficult due to the unknown values users can input into them.

The modern version of decorators, which will be officially rolled out in TypeScript 5.0, no longer requires a compiler flag and follows the official ECMAScript Stage-3 proposal. Alongside a stable implementation that follows ECMAScript standards, decorators now work seamlessly with the TypeScript type system, enabling more enhanced functionality than the original version.

With the new implementation of decorators in TypeScript 5.0, these aspects are greatly improved. Let’s take a look.

Decorators in TypeScript 5.0

TypeScript 5.0 offers better ergonomics, improved type safety, and more. Here’s a similar example of a TypeScript 5.0 decorator that overrides a class method:

function debugMethod(originalMethod: any, _context: any) {
  function replacementMethod(this: any, ...args: any[]) {
    const now = new Date(Date.now());

    console.log("start time", now.toISOString());

    const result = originalMethod.call(this, ...args);

    const end = new Date(Date.now());

    console.log("end time", end.toISOString());

    return result;
  }

  return replacementMethod;
}

class ComplexClass {
  @debugMethod
  complexMethod(a: number): void {
    console.log("DOING STUFF!");
  }
}

N.B., to try out TypeScript in an online playground, just switch the version to “nightly” or “>5.0”

With the new implementation, simply returning the function can now replace it; there’s no need for the Object.defineProperty. This makes decorators easier to implement and understand. Alongside this improvement, let’s make it completely type-safe:

function debugMethod<
  TThis,
  TArgs extends [string, number],
  TReturn extends number
>(
  originalMethod: Function,

  context: ClassMethodDecoratorContext<
    TThis,
    (this: TThis, ...args: TArgs) => TReturn
  >
) {
  function replacementMethod(this: TThis, a: TArgs[0], b: TArgs[1]): TReturn {
    const now = new Date(Date.now());

    console.log("start time", now.toISOString());

    const result = originalMethod.call(this, a, b);

    const end = new Date(Date.now());

    console.log("end time", end.toISOString());

    return result;
  }

  return replacementMethod;
}

Our decorator function in TypeScript 5.0 is greatly improved and now supports the following:

• Using generics to type a method’s arguments and return a value; the method must accept a string and a number, TArgs, and return a number, TReturn

• Typing the originalMethod as a Function

• Using the ClassMethodDecoratorContext inbuilt helper type; this exists for all decorator types

We can test to see if our decorator is truly type-safe by inspecting errors when it is used incorrectly:

Now, let’s look at an actual use case for the new TypeScript 5.0 decorators.

Decorator factory demo

We can use the type safety available in the TypeScript 5.0 decorators to create functions that return a decorator, otherwise known as a decorator factory. Decorator factories allow us to customize the behavior of our decorators by passing some parameters in the factory.

For our demo, we’ll create a decorator factory that changes the class method argument based on its own arguments. This is possible with a TypeScript type ternary operator. Our example is inspired by REST API frameworks like NestJS.

We’ll call our module rest-framework. Let’s start by creating a blank TypeScript project using ts-node:

$ mkdir rest-framework

$ cd rest-framework

$ npm init -y

$ npm install -D typescript@5.0.4 @types/node ts-node

$ touch index.ts

$ echo "console.log('Hello, world!');" > index.ts

Next, we’ll define the script to build and run the project in package.json:

{

  // ...

  "scripts": {

    "build": "tsc",

    "start": "ts-node index.ts"

  }

}

Let’s run npm start to see it in action:

$ npm start


Hello, world!

Now, let’s define our types:

interface RouteOptionsAuthEnabled {
  auth: true;
}

interface RouteOptionsAuthDisabled {
  auth: false;
}

type RouteArguments = [string] | [];

type RouteDecorator<TThis, TArgs extends RouteArguments> = (
  originalMethod: Function,

  context: ClassMethodDecoratorContext<
    TThis,
    (this: TThis, ...args: TArgs) => string
  >
) => void;

// Next, let’s define the factory decorator:

function Route<
  TThis, // The user can enable or disable auth
  TOptions extends RouteOptionsAuthEnabled | RouteOptionsAuthDisabled
>(
  options: TOptions
): RouteDecorator<
  TThis, // Do not accept a function that uses a string for an argument if auth is disabled
  TOptions extends RouteOptionsAuthEnabled ? [string] : []
> {
  return <TThis>(
    target: (
      this: TThis,

      ...args: TOptions extends RouteOptionsAuthEnabled ? [string] : []
    ) => string,

    context: ClassMethodDecoratorContext<
      TThis,
      (
        this: TThis,

        ...args: TOptions extends RouteOptionsAuthEnabled ? [string] : []
      ) => string
    >
  ) => {};
}

Now we have a route decorator that changes the class method parameter types depending on the user’s options.

Let’s create an example Route class to act as our test case:

class Controller {
  @Route({ auth: true })
  get(authHeaderValue: string): string {
    console.log("get http method handled!");

    return "response";
  }

  @Route({ auth: false })
  post(): string {
    console.log("post http method handled!");

    return "response";
  }
}

We can see that TypeScript fails to compile if we try to use authHeaderValue in the post route:

The decorator factory use case is a simple example, but it demonstrates the power of what type-safe decorators can do.

Refactoring existing decorators

If you’re using an existing TypeScript decorator, you’ll want to refactor to use the API and take advantage of its many benefits. Basic decorators can be easily refactored to the new ones, but the difference is substantial enough that advanced use cases will take effort.

For best results, follow these steps to refactor existing decorators:

• Write unit tests for your decorators

• Remove or falsify the experimentalDecorators TypeScript compiler flags

• Read this extensive summary of how the new proposal works

• Understand the limitations of modern decorators (we’ll cover this in more detail later in this article)

• Rewrite decorators using no types and use any in place of all types

• Make sure unit tests pass

• Add types

Understanding the limitations of modern decorators

The modern decorator implementation is great news for TypeScript developers, but notable features are missing. First, there’s no support for decorating method parameters. This is within the spec of the proposal, so hopefully it will be included in the final spec. Its omission is notable because popular libraries, like type-graphql, utilize this in important ways, such as writing resolvers:

class Resolver {
  @Query((returns) => Recipe)
  async recipe(@Arg("recipeId") recipeId: string) {
    return this.recipeRepository.findOneById(recipeId);
  }
}

Second, TypeScript 5.0 cannot emit decorator metadata. Subsequently, it doesn’t integrate with the Reflect API and won’t work with the reflect-metadata npm package.

Third, the --emitDecoratorMetadata flag, which was previously used to access and modify metadata for given decorators, is no longer supported. Unfortunately, there’s no real way to achieve the same functionality by getting the metadata at runtime. Some cases can be refactored. For example, let's define a decorator that validates a function’s parameter types at runtime:

function validateParameterType(
  target: any,
  propertyKey: string | symbol
): void {
  const methodParameterTypes: (string | unknown)[] =
    Reflect.getMetadata("design:paramtypes", target, propertyKey) ?? [];

  const firstParameterType = methodParameterTypes[0];

  if (typeof firstParameterType !== "string") {
    throw new TypeError("First parameter must be a string");
  }
}

We can achieve similar functionality with the improved type safety provided by TypeScript 5.0. We simply add the arguments of the method we are decorating, like so:

function debugMethod<TThis, TArgs extends [string], TReturn>(
) {

...

In theory, we could use this approach to refactor decorators that depend on getting types from Reflect for design:type, design:paramtypes, and design:returntype. This is a different way to write decorators; it is not a simple refactor because it requires using TypeScript type inference to refactor how types are acquired and validated.

Conclusion

The new decorator implementation in TypeScript 5.0 follows the official ECMAScript Stage-3 proposal and is now type-safe, making it easier to implement and understand. However, some notable features are missing, such as support for decorating method parameters and the ability to emit decorator metadata.

Basic decorators can be easily refactored to the TypeScript 5.0 version, but advanced use cases will require more effort. Developers can refactor existing decorators to use the new API and take advantage of the associated benefits. They can be less dependent on external libraries and are less likely to refactor code in the future. These changes to TypeScript's implementation of decorators are a benefit to the broader ecosystem, but community adoption could take some time.

changesets

Without using a tool, maintainers can provide checklists in pull request descriptions to remind contributors to add details to their changes. The goal is to reduce the burden and the work a maintainer has to do. This strategy is decent but reviewers miss things and it still leaves work to the maintainer to merge everything into a single release/document when multiple pull requests are merged. They could have little context as the time between merges and when a package gets released could be weeks. Maintainers want to encourage many contributions and want to make them seamless and easy. changesets is a tool created to help with the work of version management inside mono-repos. It provides a CLI interface for contributors to describe their changes in a pull request alongside the semver bump type. These bits of information are called, aptly, “changesets”. The tool is then used to perform versioning, which includes consuming all of the “changesets” since the last release, finding out the maximum semver bump type, updating the changelog and updating the appropriate internal packages.

Example

To demonstrate the capabilities and advantages of using changesets, we can see how different changes to a codebase are handled with and without it. There is an open-source e-commerce store platform that is specifically made for pet stores. The team behind the application is having a hard time organizing the code and wants to make parts of the code base more usable for different applications. They decide a mono-repo suits this well and split up pieces of code they see as generic into different packages.

Packages like ordering become adopted outside the codebase and make the lives of other developers easier by handling the ordering of items in their stores. During the course of the next week, developers on the core team and outside contributors make changes to the package. An internal developer adds a new feature that makes it easy to update the stock of an item.

// Updates the stock of an item in the database
export function updateStock(item, quantity) {}

This change requires a minor semver bump as apps or packages that get this update can safely upgrade with no breaking changes to any existing APIs.

This is not the only change that happens, an external contributor reports a critical bug where sku cannot be entered into the database when items are being inserted. sku is now a required field on the addItem function.

// Adds an item in the database
export function addItem(item, sku) {}

This change requires a major semver bump as apps or packages must make code changes to upgrade to this new version safely.

Let’s see how these two changes can be released with and without changesets.

Without changesets

After a recent release, an external developer opens a pull request to the monorepo package ordering to add a new feature:

The maintainer of the repository is pleased but notices that the contributor didn’t update the release notes in the pull request. This is outlined in the guidelines but contributors sometimes miss this which isn’t to fault them, they are focused on making changes.

The maintainer asks for the contributor to add the notes:

This change along with other changes gets merged and the maintainer proceeds to release the package a few weeks later. As part of this process, they need to figure out the semver bump type. They need to look over every pull request and see if there were any breaking changes, feature additions and/or bug fixes. They find the sku addition to addItem change and decide a full version bump is needed. As part of the release they then need to find every package which depends on the ordering package and bump it from 1.0.0 to 2.0.0, like the pet-store app. The maintainer is tasked with upgrading the app to comply with the changes. It takes a while as the contributor didn’t write much documentation for it.

Afterwards they need to create a change-log entry, by again, going through all of the pull requests merged since the last release. This alongside possibly other tasks is making the maintainer's job tedious, prone to error, and cumbersome.

With changesets

The contributor runs yarn changeset and creates a “changeset”. They describe the new updateStock function, how consuming packages can use, which package was affected by the change, and the semver bump type. They push the change and open a pull request.

The maintainer reviews the pull request and merges it, no questions asked! The “changeset” file contained everything that is needed to make the release process easy. After this, other pull requests come in and one includes a major version bump, this is due to the sku argument addition to addItem.

---
"changesets-package-ordering": major
---

Add sku argument to addItem function

The `sku` argument was added to the `addItem` function. It is a required argument due to changing business needs.

Example usage:
...

Now the maintainer is ready to release the next version of the ordering package. To do this, all they must do is run the yarn changeset version command. This command removes all of the local “changeset” files, creates the change-log entry, and automatically bumps the version of the ordering package in the package itself and in the dependants like the pet-store app.

## 2.0.0

### Major Changes

- ea13cc5: Add sku argument to addItem function

  The `sku` argument was added to the `addItem` function. It is a required argument due to changing business needs.

### Minor Changes

- 7034f8a: Add the updateStock utility function

  The `updateStock` utility function is used by applications to update the stock of an item.

Because the contributor needed to write detailed documentation for their breaking change, the maintainer has an easy time upgrading the pet-store app to comply with the changes.

With the help of changesets, the maintainer was easily able to create a release and update apps across the monorepo in a standardized and reproducible way.

Automation

We can make working with changesets even easier with automation. We always want to ensure that the correct steps are being followed by contributors and maintainers.

We can make sure contributors always create a “changeset” by installing the changesets Github Bot. This bot will comment on pull requests when one is missing:

We will then also want to ensure that no pull requests can be merged without a “changeset” by running the yarn changeset status inside of a Github Action. This can fail when contributors forget to create a changeset.

When a new version is ready to be created, maintainers can further automate this step by utilizing the changesets Github Action. This action will run the yarn changeset version command mentioned earlier and create a pull request with all of the file changes it creates.

In Conclusion

Using the changesets package, many of the steps which make the release process a burden on maintainers can be lifted. Contributors can be reminded to make detailed notes about their changes and maintainers can easily create new versions of packages using a single CLI command. With high-quality and recurring package releases, consumers will have an easy time upgrading to new versions.

Ruck

Ruck is a minimal framework for building React apps with Deno. It leans into Deno-specific features like ES Modules and import maps which makes it a great showcase for the new runtime. It doesn’t use a bundler so it does not support writing React components in JSX and all configuration defines itself in code. Using createElement everywhere is not the best developer experience. I could see another framework adopting Ruck under the hood to solve a lot of the problems for it. Ruck is what you are looking for if you want a framework in which you have control of what is going on and don’t like the “magic” of other frameworks.

An example component is written with Ruck:

import { createElement as h } from "react";
import useOnClickRouteLink from "ruck/useOnClickRouteLink.mjs";
import useRoute from "ruck/useRoute.mjs";

export const css = new Set([
  "/components/ExampleComponent.css",
]);

export default function ExampleComponent({ href, children }) {
  return createElement("a", 
        { className: "NavLink__a", href, onClick: () => console.log('Hello World!') },
        children
    );
}

Aleph.js

Aleph.js is a full-stack web framework for building React apps with Deno. Second, to Fresh, it is the most popular Deno-native React framework. It leans into Deno for some of its features but also provides much more. Aleph.js is inspired by Next.js even giving some of the same syntaxes for some features. Aleph.js supports server-side rendering as well as static-site generation, creating standalone APIs, file-base routing, and React Hot Module Reloading. To support separate file types such as JSX and CSS it doesn’t use webpack but instead uses esbuild.

An example component is written in Aleph.js:

import React from 'react';
import Logo from '../components/logo.tsx'

export default function ExampleComponent() {
  return (
    <div>
      <Logo />
      <h1>Hello World!</h1>
    </div>
  )
}

Similarities

There are similarities between Ruck and Aleph.js. One of those similarities is the support for import maps. Without the usage of NPM or another package manager, Deno depends on HTTP imports. This means imports usually look like this:

import React from "https://esm.sh/stable/react@18.2.0/es2021/react.js”;

Deno recommends putting all module imports into a single deps.ts file to be re-exported. The issue with this approach is that imports are still not compatible with Node.js/webpack counterparts. A better way (and browser-compliant way) to do this is with import maps. Import maps are a recent browser feature that instructs the browser where dependencies for a module are located.

An example import map:

{
  "imports": {
    "react": "https://esm.sh/stable/react@18.2.0/es2021/react.js",
  }
}

A component that uses the import map:

import React from "react";

export default function ExampleComponent() {
  return <div />;
}

To use an import map in Aleph.js, one needs to define one file named import_map.json in the root directory. Using one in Ruck is also simple, define the file and pass it into Deno at runtime:

deno run \
    --allow-env \
    --allow-net \
    --allow-read \
    --import-map=importMap.json \
    scripts/ruck-serve.mjs

The issue with import maps is that browser support is still poor with Safari and Firefox not supporting it out of the box. The good news is that Ruck uses a shim to provide support for older browsers.

Another similarity is their focus on server-side rendering (SSR) React components. SSR can provide performance, SEO and other benefits over client-side rendering. If a React component depends on fetched data, opting to do so on the server means a component can render before sending data to the client. This means no loading states to show to the user and generally better performance. Ruck supports data fetching on the server at a component level whereas other frameworks usually only support this at the page level. Aleph.js lets you define a ssr function inside a page component file to achieve this. Aleph.js also supports a special hook, useDeno, for use in a component.

Example of using useDeno to fetch data on the server side in Aleph.js:

import React from 'react'
import { useDeno, useRouter } from 'aleph'

export default function Post() {
  const { params } = useRouter()
  const post = useDeno(async () => {
    return await (await fetch(`https://.../post/${params.id}`)).json()
  })

  return (
    <h1>{post.title}</h1>
  )
}

When it comes to styling your React app with CSS, both Ruck and Aleph.js support component-level CSS imports. This allows for sending CSS to the browser that requests it (i.e. when a component renders). Ruck allows for this via an exported component variable named css. You can achieve the same behavior in a variety of ways with Aleph.js but the recommended approach is to use CSS modules.

Example of using the css function in Ruck:

import React from 'react'
import Heading, { css as cssHeading } from "./Heading.mjs";
import Para, { css as cssParagraph } from "./Para.mjs";

export const css = new Set([
  ...cssHeading,
  ...cssParagraph,
  "/components/ExampleComponent.css",
]);

export default function ExampleComponent() {
   ...
}

Example of using a CSS module in Aleph.js:

import React from 'react'
import styles from './exampleComponent.module.css'

export default function ExampleComponent() {
  return (
    <>
      <h1 className={styles.title}>Hi :)</h1>
    </>
  )
}

A perk of being a server-side rendered application is having access to the HTTP request during the rendering lifecycle. This can be helpful if you need to access headers or change the response. With Ruck, the HTTP response is available in a React context, TransferContext. In Aleph.js we can use the ssr function.

Example of modifying the HTTP response in Ruck:

import React from 'react';
import TransferContext from "ruck/TransferContext.mjs";

export default function PageError({ errorStatusCode, title, description }) {
  const ruckTransfer = useContext(TransferContext);

  if (ruckTransfer) ruckTransfer.responseInit.status = errorStatusCode;

  ...
}

Example of modifying the HTTP response in Aleph.js:

import React from 'react';
import { useDeno } from 'aleph';

export default function ExampleComponent() {
  const isLoggedIn = useDeno(req => {
    return req.headers.get('Auth') === 'XXX'
  }, { revalidate: true })

  return (
    <p>isLoggedIn: {isLoggedIn}</p>
  )
}

Differences

There are notable differences between the two frameworks to be aware of. Popularity and developer experience are the two largest. Ruck is new so doesn’t have the community backing that a framework like Aleph.js has. By looking at Deno X Ranking, Aleph.js is the second most popular React framework by Github star count with 4.8k compared to Ruck with only 120. Star Count isn’t the best metric but it gives you a good idea about developer intent.

Ruck will favor the developers who like a high level of control over exactly how their application functions. Ruck has configuration set in code, for example routing you must define yourself while Aleph.js handles this for you. Aleph.js can be run with zero configs and has project templates to get developers started. You can opt-in to features based on config. In Ruck, you must spend time setting up the basics of the application yourself.

Static websites are desirable if your web application has all the data it needs at build time. This can simplify deployments as there needs to be no running Deno server. Place the built folder of HTML, CSS and JS to a deployment target like Github Pages or Cloudflare. Aleph.js supports static-site generation which is helpful for these situations while Ruck does not. Like getStaticPaths in Next.js, you can define a paths key in the ssr function inside a component file to specify the paths this route can handle:

import type { SSROptions } from 'aleph/types';

export const ssr: SSROptions = {
  paths: async () => {
    const posts = await (await fetch('https://.../api/posts')).json()
    return posts.map(({ id }) => `/post/${id}`)
  }
}

And then run aleph build , simple as that.

Final Thoughts

With the popularity of Deno continuing to increase, Ruck and Aleph.js are two Deno-based React web frameworks catering to two different sets of developers. Ruck being a newcomer, doesn’t have the same level of polish Aleph.js has but offers more control. Aleph.js offers a great developer experience with zero config needed and lots of powerful features. These minimal frameworks bake in a lot of built-in modern browser features which can lead to a minimal and lean tech stack that contrasts a lot of the complexity in the frontend ecosystem seen today. Deno’s large amount of built-in features leads to less work being done by third-party tools. React frameworks can focus on developing innovative and interesting new features while developers are at ease knowing they made a great choice for their web application tech stack.

Using API network requests is easily one of the slowest steps required to render pages, and a slow-running app can mean a poor user experience. Having great performing pages can also improve your search engine optimization (SEO) dramatically.

If you own the API and know how to make it faster, great! But if not, you don’t have control over the performance and speed of your app. You may even be using server-side generation, which passes data fetching and page rendering to the server, and yet, your app still may not be fast enough. This is where static site generation comes into play. There are different ways to implement SSG, but first an explanation into how it works.

What is static site generation?

One of the ways software developers can optimize their applications is by doing the work ahead of time or by using caching.

Static site generation is the process of building pages (pre-rendering) into static assets and serving them to users instead of doing it per request, especially when our data is static or doesn’t change often. We can also create as many builds as we want, and most of this work can be done on a hosted server, making the process easy.

Thankfully, there are many static site generation tools based on React.js, with some of the most interesting ones being Gatsby, Next.js, and Remix.js. All three achieve serving performant web applications in their own way.

For example, Gatsby has a great ecosystem of plugins to get you started quickly. Next.js is flexible, allowing you to opt into SSG on a per-page basis. Remix.js, while it doesn’t currently support SSG in the traditional sense, strives for page performance through caching assets on edge servers.

After understanding how Gatsby, Next.js, and Remix achieve performance through caching and SSG, choosing the best one for the job becomes easier, and we can take the techniques used and apply them to other frameworks and tools.

Using Gatsby for static site generation

Gatsby is a React-based framework made for building statically generated web applications. It is one of the most popular ways developers perform SSG on the web today and has a huge community and plugin ecosystem.

Here’s how it works. Developers write functions in Node.js to fetch data, create pages, and fill them with content. Matching these pages with React components (Gatsby calls these “templates”) is how Gatsby knows what to render and when.

Builds can be created on a developer’s local machine but also on many SaaS hosting solutions like Gatsby Cloud, Netlify, and others. Once a build is complete, it can then be deployed to a CDN for ultra-fast serving to users.

One of the best aspects of Gatsby is that its developer ecosystem is extensive. It has data-source plugins that make it incredibly easy to fetch data from an external API (take Shopify, for example) and image optimization plugins, just to name a few.

Gatsby seems to have a plugin for everything — over 2,000 and counting! Not only does it have several plugins to choose from, but the documentation is extensive and there are many starter templates to take inspiration from.

With Gatsby, you get a very quick website by default while you’re developing at a quick rate. If you are building a frontend for a CMS or creating a commerce backend, Gatsby is a great choice.

Here is an example of the code used to generate pages in Gatsby:

exports.createPages = async function ({ actions }) { const { data } = await graphql(query { allMarkdownRemark { nodes { fields { slug } } } }) data.allMarkdownRemark.forEach(node => { const slug = node.fields.slug actions.createPage({ path: slug, component: require.resolve(./src/templates/blog-post.js), context: { slug: slug }, }) }) };

Of course, all frameworks have their downsides, and Gatsby is no exception to this rule. Here are some cons to using Gatsby.

Cons to using Gatsby for static site generation

• Server-side rendering (SSR) is relatively new to Gatsby, so the docs aren’t thorough
• The build process is tough to debug
• Devs must use GraphQL to get data into React components, which may make the learning curve steep for some
• Many aspects of Gatsby, including data fetching, are Gatsby-framework specific, so it’s challenging to switch to a different framework in the future
• Builds can become slow depending on the amount of data processing you need — like most SSG-based technologies, the more your data increases, the number of pages to create and images to process also increase

Considering the trade-offs, however, Gatsby can still be a great choice for your project.

Static site generation with Next.js

Next.js is a React-based framework for building both dynamically and statically rendered web applications. It’s very popular and has many developers using it daily.

There is now a relatively new feature that added SSG support to Next.js, with the unique proposition that it can be done on a per-page basis. The build process is similar to traditional SSG frameworks like Gatsby, but you can choose the pages the SSG occurs on.

Developers also have the choice of using incremental static generation (ISR), which is an advanced method that builds pages at runtime if they are not present or need to be rebuilt due to developer-set invalidation.

If you have only chosen to use SSG for your pages, deployment is similar to Gatsby and can be done with a variety of vendors. It uses Node.js to fetch data and build your pages. If you use any of the SSG unsupported features, such as using a server to fetch data and build pages at runtime, you’ll have a more involved deployment process.

One of the nicest aspects of Next.js is that it gives you the freedom to do what you want. As mentioned, you get the choice on a per-page basis to use SSG, SSR, or the new method, ISR. Better yet, it doesn’t prescribe anything like GraphQL when it comes to getting the data into your React components.

Here is an example of the code required for Next.js to statically generate a page:

function Blog({ posts }) { return (

{posts.map((post) => (
• {post.title}
))}

export async function getStaticProps() { const res = await fetch('https://example.com/posts') const posts = await res.json()

return { props: { posts, }, } }

export default Blog

While Next.js is a great option, it’s not for everyone.

The cons to using Next.js for static site generation

• The freedom it allows can be daunting for new developers who don’t have much experience writing web applications
• Unlike Gatsby, there aren’t any out-of-the-box plugins, so there is boilerplate code you may need to write if the data you are fetching is from a popular vendor like Shopify or WordPress
• The deployment and infrastructure can be cumbersome if you mix and match features and don’t use the Vercel, the company behind Next.js, deployment solution.

However, if you want the freedom Next.js provides and don’t mind using Vercel’s deployment platform, Next.js is a great choice for web applications that need SSG.

Using Remix.js with modern React tools

Remix.js is a React-based framework for building primarily dynamically rendered web applications. It’s a very new framework to many, as it was closed-source until recently.

Remix is interesting because it does not provide an option to pre-compile pages into a static asset bundle as Gatsby or Next.js do. There is no concept of SSG or “builds”, data-fetching is on-demand by default and HTML is compiled on the server per request (SSR).

Instead Remix depends on distributed computing to perform the same features one would get with SSG. The Remix server is compatible with edge computing vendors, such as Cloudflare Workers and Fly.io.

The server that fetches the data and compiles the HTML is deployed to these edge servers close to your users, resulting in much faster response times than a traditional web application setup. Our data, by default, is still fetched per request, which may be still too slow for your needs. The Remix teams recommends you create custom cache code for this purpose.

There are edge server compatible databases that are usually in-memory, like SQLite, Redis, and LRU caches. When the data changes, you can evict data inside these caches manually or automatically, just like you would trigger a new build in a traditional SSG setup.

It’s also recommended to use HTTP caching and serve static content on a CDN to serve web pages where the data doesn’t change often. This means you’ll get similar performance to a framework that uses SSG without it being coupled to a specific framework or having to worry about builds.

Like Next.js, Remix gives the developer more freedom than prescribed frameworks like Gatsby when it comes to providing a fast web application experience. One of the great aspects of Remix.js is that it relies heavily on platforms like edge computing and browsers to accomplish most of its features.

This means when you learn Remix, you learn more about the web, not how to develop in a specific framework. There is a surprisingly large amount of deployment options for Remix despite being new, so you can shop around for the best one.

Here is an example of a React component that fetches data and compiles the HTML on an edge server and displays it on the page:

export const loader: LoaderFunction = async ({ request }) => { const userId = await requireUserId(request); const noteListItems = await getNoteListItems({ userId }); return json({ noteListItems }); };

export default function NotesPage() { const data = useLoaderData() as LoaderData;

return data.noteListItems.map((note) => (

The cons of using Remix

• Like Next.js, the freedom Remix provides can be daunting for developers just starting, but the documentation is great so far, so you can learn what it has to offer
• Creating custom caching mechanisms for your web content could result in boilerplate code, which isn’t great
• The framework is still new and just now providing examples of how to perform data-caching on the server

In general, Remix is a great option if you aren’t afraid to take on a relatively new option in the space.

Conclusion

There is plenty to consider when choosing between Gatsby, Next.js, and Remix.js to build your application. Which one you should choose depends on your ideal setup, your experience level, the vendor you want to host on, and how much code you want to write. No matter what you choose, all three options provide ways to serve web pages fast to your users.

What is Deno?

Deno incorporates a few other aspects, such as a built-in TypeScript compiler, linter, formatter, and package manager. It also supports ESM modules, uses the web platform as a base for its standard library, and has a secure-by-default design around IO.

This results in a drastically simplified development experience when compared to Node.js. I started using Deno for a CLI tool I needed to write (using deno-cliffy) and was pleased with the experience. When I first heard about Denoflare, I thought it would be the ideal opportunity to experiment with Cloudflare Workers while gaining more experience with Deno.

Using Deno with Cloudfare Workers

Cloudflare Workers is an alternative to serverless infrastructure when compared to services like AWS Lambda. At its core, it acts like a serverless function, taking the request information, applying logic, and sending a response.

The expected use-case is different in that it’s meant to act as a middleware, intercepting requests sent to an origin server, and applying logic. Cloudflare behaves similarly to browser service workers.

Cloudflare Workers is not full of isolated containers being spun up, providing the benefit of zero cold-starts. Some other nice features of Cloudflare Workers are global CDN deployments and paying on a per-request basis.

What is Denoflare?

Denoflare is a small framework that lets you publish Cloudflare Workers written in Deno. It’s a natural fit, as both Deno and Cloudflare Workers follow standardized web platform runtime APIs.

Denoflare lets you serve your worker locally to test your changes in an isolated environment that is similar to how Cloudflare would run them. It supports a great developer experience with hot-reloading, being able to publish the worker right to the Cloudflare platform, tailing logs in production, and more.

This has a similar experience to using another worker framework like Miniflare, except it's much simpler because Deno is doing most of the work. For example, instead of depending on Jest as Miniflare does, you can write tests to run with the native Deno test runner.

Working with Cloudflare and Denoflare

The best way to experience Cloudflare and Denoflare is by going through a real-world example use case. Using minimal code, we will set up A/B testing for a blog. Half the time, the user will be put into the test group and will see a new header. The others will be put into the control group and will not see the new header.

Using Cloudflare Workers, this is as simple as intercepting requests to the blog origin server, placing the user into a group based on our split, and setting the response header Set-Cookie with the group name. After this is done, our blog can read the cookie to decide which header to show.

We are omitting the code needed to change the header in the blog as there are several different ways you can do so.

Setting up your Cloudflare Workers account

The first thing we must do is sign up for a Cloudflare account with Cloudflare Workers set up. Once you are done, create a free .dev subdomain for testing our worker on. We will choose the free plan.

Creating a free .dev subdomain for testing.

Note: You can change this later to be the domain you would use in production.

Denoflare requires an API token to allow us to push our compiled worker to Cloudflare. Go to the tokens page in the Cloudflare dashboard and select Create Token. Once there, you should choose Edit Cloudflare Workers as the template. Note this token for later.

From the overview screen, copy the account ID for use later.

Developing with Denoflare

Deno provides a great development experience, which we will keep intact when using Denoflare.

Note: You can find all the code in this GitHub repo.

First, set the Deno version and install it on your local machine using asdf:

echo "deno 1.16.0" > .tool-versions && brew install asdf && asdf plugin add deno && deno install

Let’s configure our IDE. If you are using Visual Studio Code, you will have a great experience with Deno. Install the Deno extension, which can be found here. It enables type-checking, linting, and more.

To enable it, you must turn it on in the workspace settings file at .vscode/settings.json:

{
  "deno.enable": true
}

Next, we will install Velociraptor, a script manager for Deno. Velociraptor makes it easy for all developers to use the same scripts when performing common tasks (think npm scripts).

Run this command in your console:

deno install -qAn vr https://deno.land/x/velociraptor@1.3.0/cli.ts

We will define a Velociraptor script to invoke the Denoflare CLI, allowing us to run Denoflare commands such as serve and push.

{
  "scripts": {
    "denoflare": "deno run --unstable --allow-read --allow-net --allow-env https://raw.githubusercontent.com/skymethod/denoflare/v0.3.3/cli/cli.ts"
  }
}

Now that we have our IDE and environment runtime setup, we can move on to the code. With Deno, it’s common to declare all dependencies in a single file named deps.ts. This is because modules are defined by URLs and can become difficult to manage if scattered everywhere in a project.

We need one type of Cloudflare Workers, which Denoflare defines:

export type { IncomingRequestCf } from "https://raw.githubusercontent.com/skymethod/denoflare/v0.3.0/common/cloudflare_workers_types.d.ts";

Once we have this type, we can write our A/B tester logic in index.ts:

import { IncomingRequestCf } from "./deps.ts";

/**
 * Based on the A/B Testing Cloudflare Worker example.
 * Ref: https://developers.cloudflare.com/workers/examples/ab-testing
 */
function fetch(request: IncomingRequestCf): Response {
  const NAME = "experiment-0";

  const TEST_RESPONSE = new Response("Test group");
  const CONTROL_RESPONSE = new Response("Control group");

  // Determine which group this requester is in.
  const cookie = request.headers.get("cookie");
  if (cookie && cookie.includes(`${NAME}=control`)) {
    return CONTROL_RESPONSE;
  } else if (cookie && cookie.includes(`${NAME}=test`)) {
    return TEST_RESPONSE;
  } else {
    // If there is no cookie, this is a new client. Choose a group and set the cookie.
    const group = Math.random() < 0.5 ? "test" : "control"; // 50/50 split
    const response = group === "control" ? CONTROL_RESPONSE : TEST_RESPONSE;
    response.headers.append("Set-Cookie", `${NAME}=${group}; path=/`);

    return response;
  }
}

export default {
  fetch,
};

The last step is to declare a .denoflare file, which Denoflare uses to run and publish your Cloudflare Worker:

{
  "$schema": "https://raw.githubusercontent.com/skymethod/denoflare/v0.3.3/common/config.schema.json",
  "scripts": {
    "a-b-test-local": {
      "path": "index.ts",
      "localPort": 3030
    }
  },
  "profiles": {
    "account1": {
      "accountId": "INSERT_ACCOUNT_ID_FROM_PREVIOUS_STEP",
      "apiToken": "INSERT_API_TOKEN_FROM_PREVIOUS_STEP"
    }
  }
}

That’s it! Let’s serve the Cloudflare Workers locally to make sure everything is working properly:

vr denoflare serve a-b-test-local

This should be the output:

Compiling https://raw.githubusercontent.com/skymethod/denoflare/v0.3.3/cli-webworker/worker.ts into worker contents...
Compiled https://raw.githubusercontent.com/skymethod/denoflare/v0.3.3/cli-webworker/worker.ts into worker contents in 277ms
runScript: index.ts
Compiled index.ts into module contents in 142ms
worker: start
Started in 456ms (isolation=isolate)
Local server running on http://localhost:3030

Now, open a browser and go to localhost:3030. You will see that the response is the group the user is put into:

Testing the application

Because Cloudflare Workers is a Deno application, we can use its tools to validate and test our code. All of these run automatically as you are developing in Visual Studio Code, thanks to the extension you installed.

Run the Deno linter with:

deno lint

Make sure the Deno app compiles with:

deno compile index.ts

Write a few tests and run them with:

deno test index.test.ts

Deploying Cloudflare Workers

You should have no trouble deploying Cloudflare Workers once you have the correct account ID and token in the Denoflare configuration (.denoflare).

By default, Denoflare pushes your worker to the .dev subdomain you have set up for your account. Deploy it to the Cloudflare instance by running:

vr denoflare push a-b-test-local

You should see the worker appear instantly on your worker dashboard.

By default, its public access is disabled. Go to the service details page and enable the route.

When you go this route, you should see a response with what group this request gets put into.

Conclusion

Denoflare is a simple mini-framework built around Deno that allows us to easily publish Cloudflare Workers.

Due to the way it implements web standards API and the similarity to Cloudflare Workers in the security model, Deno is a natural fit. Cloudflare Workers is a powerful way to deploy middleware logic to the cloud close to where users are using your applications due to their edge CDN strategy, where they deploy workers around the globe.

Have a look at other examples the Cloudflare team put together so you can get an even better idea of what else it can be used for.

How come? Because GraphQL is an opinionated API spec where both the server and client buy into a schema format and querying format. Based on this, they can provide multiple advanced features, such as utilities for caching data, auto-generation of React Hooks based on operations, and optimistic mutations.

Sometimes libraries can be too opinionated and offer too much "magic". I’ve been using Apollo Client for quite some time and have become frustrated with its caching and local state mechanisms.

This “bloat,” along with recently seeing how mismanaged the open-source community is, finally broke the camel's back for me. I realized that I needed to look elsewhere for a GraphQL client library.

What is urql?

Enter urql, which is a great alternative. It isn’t the new kid on the block — it’s been around since 2019 — but I’ve just made the switch and stand by my decision.

Most of the lingo is the same as Apollo Client, which made switching from Apollo to urql fairly straightforward. urql has most of the same features but also offers improvements, including better documentation, better configuration defaults, and first-party support for things like offline mode, file uploads, authentication flows, and a first-party Next.js plugin.

When you stack Apollo Client and urql against each other, you’ll start wondering why Apollo Client has been so popular in the first place.

Bye Apollo Client 👋, hello urql

As I'm writing this, the Apollo Client Github repository issue count stands at 795. In comparison, urql has 16. “But issue count doesn't correlate to code quality!" is what you may say to me. That’s true, but it gives you the same feeling as a code smell — you know something isn't right.

Looking deeper, you can see a large amount of issues open, bugs taking months to fix, and pull requests never seem to be merged from outside contributors. Apollo seems unfocused on building the great client package the community wants.

This sort of behaviour indicates to me that Apollo is using open-source merely for marketing and not to make their product better. The company wants you to get familiar with Apollo Client and then buy into their products, not truly open-source software in my opinion. This is one of the negatives of the open-core business model.

I started to look elsewhere for a GraphQL Client that had a more happy and cohesive community. When a tool is designed well and with features the community wants, fewer issues are created and there is less of a need for pull requests. Formidable is the agency behind urql, and they care about creating applications in fast and maintainable ways, compared to trying to funnel users into using their products.

Why use urql?

For me, urql is a breath of fresh air after working with Apollo Client for so long. There are a lot of little things that add up to a much better developer experience, especially for newcomers. Here are just a few.

Documentation in urql is thorough Having great documentation is a key feature for any open-source library. Without great docs, there will be more confusion among the community over how to use it and how it works internally. I attribute urql’s thorough docs to why it has such a low issue count. It only took me a few hours to read the entire documentation.

This is impressive because it shows how focused the library is and how thought-out the structure is. Some of the highlights include this one-pager on the architecture of how urql works and this table comparing itself to other GraphQL clients (like Apollo).

Plugins and packages have first-party support in urql urql really caught my attention when I heard it had first-class support for additional functionality such as offline mode, file uploads, authentication, and Next.js. These are all features that I've always thought of as basic for a GraphQL client, and it's great to see urql have first-party support for them.

For instance, the urql authentication exchange package has you implementing only a few methods to have an entire authentication flow within your client, including token refresh logic. You can achieve all of these things in Apollo Client, but there are no official docs or packages. This means you spend more time to research community solutions, hacks, and code.

// All the code needed to support offline mode in urql
import { createClient } from "urql";
import { offlineExchange } from "@urql/exchange-graphcache";
import { makeDefaultStorage } from "@urql/exchange-graphcache/default-storage";

const storage = makeDefaultStorage({
  idbName: "apiCache",
  maxAge: 7, // The maximum age of the persisted data in days
});

const cache = offlineExchange({
  schema,
  storage,
  updates: {
    /* ... */
  },
  optimistic: {
    /* ... */
  },
});

const client = createClient({
  url: "http://localhost:3000/graphql",
  exchanges: [cache],
});

It's also great that I haven't had to give up things I loved when working with Apollo Client, such as the dev tools and React hooks generation because urql has a dev tools browser extension and a plugin for graphql-code-generator.

Caching in urql is easy and effective There is a common developer motto that cache invalidation is one of the hardest things in programming. After many hours debugging Apollo Clients normalized cache, I believe it. urql's caching defaults are sensible to the newcomer and can be extended to become more advanced.

I appreciate that it doesn't force you to use a normalized cache by default, but comes with a document cache instead. This works by just hashing the query and its variables — it’s simple and effective!

Learning how a complex, fully normalized caching store works just to get started using a client library seems heavy handed. Only offering normalized caching is something I felt Apollo Client got wrong.

There is a steep learning curve to managing a normalized cache, and it's unnecessary for many applications. It's fantastic that urql offers this as a separate package that you can opt into at a later time. I’ve seen this trend demonstrated with other packages as well such as React Query.

While a vast majority of users do not actually need a normalized cache or even benefit from it as much as they believe they do. - React Query Docs

import { ApolloClient, InMemoryCache } from "@apollo/client";

const client = new ApolloClient({
  uri: "http://localhost:4000/graphql",
  // Normalized cache is required
  cache: new InMemoryCache(),
});

import { createClient } from "urql";

// Document cache enabled by default
export const client = createClient({
  url: "http://localhost:4000/graphql",
});

Local state is simplified in urql urql stays true to server data and doesn't provide functions to manage local state like Apollo Client does. In my opinion, this is perfectly fine as full-on libraries to manage local state in React are becoming less needed. Mixing server-side state and local state seems ideal at first (one place for all state) but can lead to problems when you need to figure out which data is fresh versus which is stale and when to update it.

React Context is a great solution for situations where you have lots of prop drilling going on, which is sometimes the main reason people reach for a local state management library. I would also recommend XState if you are looking for a way to manage stateful workflows, which sometimes people use Redux reducers for.

Understandable default behavior with Exchanges Exchanges are similar to links in Apollo Client and offer ways to extend the functionality of the client by intercepting requests. The difference with urql is that you can opt into even the basic ones, allowing you more control and understanding over the behaviour of the client.

When getting started, the client has no required exchanges and uses a default list. In my experience, starting off with just a few exchanges and adding more as time went on or when I needed them made debugging easier. urql shows that it takes extensibility seriously in supporting many different use-cases.

Here is an example of the exchanges you might use after you get used to urql:

import {
  createClient,
  dedupExchange,
  cacheExchange,
  fetchExchange,
} from "urql";

const client = createClient({
  url: "http://localhost:4000/graphql",
  exchanges: [
    // deduplicates requests if we send the same queries twice
    dedupExchange,
    // from prior example
    cacheExchange,
    // responsible for sending our requests to our GraphQL API
    fetchExchange,
  ],
});

uqrl offers a Next.js support plugin Next.js is one of the most popular ways to use React these days. Integrating Apollo Client to use Next.js SSR in the past has always been a huge pain. With every upgrade, you will have to look for examples and likely need to change how it works.

With no official plugin from Apollo, you will have to keep maintaining this integration. As mentioned previously, urql has an official plugin for Next.js. This makes it easy to integrate.

// Simple React component integrating with Next.js using the plugin
import React from "react";
import Head from "next/head";
import { withUrqlClient } from "next-urql";

import PokemonList from "../components/pokemon_list";
import PokemonTypes from "../components/pokemon_types";

const Root = () => (
  <div>
    <Head>
      <title>Root</title>
      <link rel="icon" href="/static/favicon.ico" />
    </Head>

    <PokemonList />
    <PokemonTypes />
  </div>
);

export default withUrqlClient(() => ({
  url: "https://graphql-pokemon.now.sh",
}))(Root);

Conclusion urql has advantages over Apollo Client when it comes to its unified community, great documentation, and first-party plugins and caching system. I especially like how they seem to be working and engaging with the community instead of against it.

I’ve been trying a lot of GraphQL clients lately to see what else is out there to compare them to Apollo and it's been refreshing to see how great urql is. I foresee myself using it going forward for all my GraphQL apps. I hope this prompts you to try out urql for yourself and see what you think. Thanks for reading!

So what tools can you add to your workflow to supercharge development in React? Storybook.

What is Storybook?

Storybook is, according to its website, an “open-source platform that allows you to document, view, and test many permutations of your JavaScript components within an isolated environment.”

Before I start to create a component, I first create stories for it in Storybook, then I start integrating it into my React app. This leads to writing more code, but leads me to reduce my churn.

It also forces me to think about edge cases, how the API of a component should be defined, and to decouple it from my main application.

It's similar to Test-driven Development: write out the test cases before the code is written, but in this case, the tests are stories of all the states a component can be in. Being its own app, iterating through component designs is quick. Storybook has led to me ironing out edge cases, catching more bugs, and ultimately, finishing features faster.

Integrating Storybook with visual testing services like Percy can help your team move fast with each pull request showing you diffs of new component changes. You can also further test components that are API-driven by mocking out query responses with Mock Service Worker.

Setting up components in Storybook

Let's go through an example of how to get features done quicker using Storybook. Stories are just components rendered with a particular set of props. You can have as many or few as you like.

Let's pretend we are building a blog and want to have a list of entries on the index page. Let's make a story for each state of the component which will render each entry.

import React from "react";

import { BlogEntryListItem } from "./BlogEntryListItem";

export default {
  title: "BlogEntryListItem",
  component: BlogEntryListItem,
};

export const BlogEntryListItemLoaded = () => (
  <BlogEntryListItem
    title={"A Fake Blog Post Title"}
    excerpt={"Lorem Khaled Ipsum is a major key to success."}
    date={"2019-01-01"}
    lastUpdatedAt={"2020-01-02"}
    slug={"/a-fake-blog-post-title"}
  />
);

export const BlogEntryListItemLongExcerpt = () => (
  <BlogEntryListItem
    title={"A Fake Blog Post Title"}
    excerpt={
      "Lorem Khaled Ipsum is a major key to success. You should never complain, complaining is a weak emotion, you got life, we breathing, we blessed. The key to success is to be yourself."
    }
    date={"2019-01-01"}
    lastUpdatedAt={"2020-01-02"}
    slug={"/a-fake-blog-post-title"}
  />
);

export const BlogEntryListItemLoading = () => <BlogEntryListItem loading />;

We focus on the API of the component before coding the real thing. I like to mirror product requirements here.

In this example, I knew some blog entry excerpts would be long, so I created a story for it. I also needed to have a loading state because I planned to use react-loading-skeleton.

The next step is creating the basic code for the component:

import React from "react";

export const BlogEntryListItem = (props) => {
  if (props.loading) {
    return <div>Loading...</div>;
  }
  return (
    <NoColorLink to={props.slug}>
      <BlogListItemWrapper>
        <Description>
          <Title>{props.title}</Title>
          <span>
            Published: {props.date}
            <br />
            Last Updated: {props.lastUpdatedAt}
          </span>
          <Excerpt>{props.excerpt}</Excerpt>
        </Description>
      </BlogListItemWrapper>
    </NoColorLink>
  );
};

Here’s what it looks like in Storybook:

The great thing about this is that we haven't touched our main application at all. We didn't have to muck around with production configuration, environment variables, or running local API services.

Improving components with Storybook

Defining all the states the component needs and writing a simple implementation has us looking great so far!

Without adding our BlogEntryListItem component to to the main application, we can start making improvements right away. As you probably noticed, the excerpt is quite long and wraps the inside the <div>, so let's fix that using overflow: hidden.

Look! We improved our component without even stepping foot into our main app. We can go even further using some add-ons that ensure our component is even more capable.

One of the add-ons that Storybook comes with by default is Storybook Viewport Addon, which allows you to see what your components look like on various screen sizes.

Using this add-on in this example shows us that we can’t read excerpts on mobile.

You can see how using Storybook can improve our components without ever needing to run our main React application. This is the true power of working with components.

Improving speed with Storybook

When iterating through components, many visual changes are bound to happen. Having a coworker pull your code changes and run Storybook locally to see changes works is slow, and we can certainly work faster.

Visual Testing tools give you screenshots of the visual diff between components as you iterate. For example, a tool can generate a screenshot of a fix for our component to properly render entries on mobile.

This works through a Continuous Integration service like CircleCI or Github Actions, you can build Storybook and use the Percy Storybook plugin to snapshot all of your stories. It renders every story in a consistent browser environment and sends the HTML over to Percy for it to render. It then compares these rendered stories to previous builds to mark differences, like this:

Percy provides a great Github Action, which does all of this automatically. Here is an example pull request which implements this.

In my experience, using visual testing with Storybook has caught many regressions by spotting changes which we didn't catch in code review.

Mocking out API queries with Storybook

Not only can Storybook provide us with a way to test components’ look and feel, but it can also can help us test behavior. Some components in your application most likely query data from a remote API. These are most often called "container components" or "page components.”

Providing fake data for your components is great, but we can get closer to reality by mocking the API requests that the components perform.

This example uses a REST API but the libraries used are compatible with GraphQL.

Thinking back to our blog entries, typically a parent component would query for a bunch of entries:

import React from "react";
import { useQuery } from "react-query";

import { BlogEntryListItem } from "./BlogEntryListItem";

async function fetchBlogEntries() {
  const res = await fetch("<https://fake-blog-entries-url.com>");
  if (!res.ok) {
    throw new Error(res.statusText);
  }
  const data = await res.json();
  return data.results;
}

export const BlogEntries = (props) => {
  const { status, data, error } = useQuery("blog-entries", fetchBlogEntries);
  return data.map((datum, index) => {
    return <BlogEntryListItem key={index.toString()} {...datum} />;
  });
};

It would be nice if we could mock a response from the server in Storybook to see how the component behaves in different scenarios. There is a great library called Mock Service Worker that will intercept browser network queries and provide mock responses. Coupled with the Storybook add-on for this module, we can provide mock data:

import React from "react";
import { QueryClient, QueryClientProvider } from "react-query";
import { rest } from "msw";

import { BlogEntries } from "./BlogEntries";

const mockedQueryClient = new QueryClient({
    defaultOptions: {
    queries: {
        retry: false,
    },
    },
});

export default {
    title: "BlogEntries",
    component: BlogEntries,
};

export const BlogEntriesStates = () => (
    <QueryClientProvider client={mockedQueryClient}>
    <BlogEntries />
    </QueryClientProvider>
);

BlogEntriesStates.story = {
    parameters: {
    msw: [
        rest.get("<https://fake-blog-entries-url.com>", (req, res, ctx) => {
        return res(
            ctx.json({
            results: [
                ...
            ],
            })
        );
        }),
    ],
    },
};

Conclusion

I covered a lot here, so I'll summarize my Storybook workflow:

1. Receive requirements from product
2. Think about my component hierarchy
3. For each component, write a story for each significant state
4. For each "page" or feature, write a story and add API mocks
5. Write the code for each component which satisfy each state
6. Use Visual Testing in CI to test my changes against the main branch

Notice that there are many steps, and it can take some time to adapt to this new flow. But after practicing this workflow, it seems natural to me now and I would never go back to writing React code without using Storybook alongside me.

Storybook is a perfect way to prototype components and make sure visual components get the love they deserve.

A Bit of Background

I ran into this problem when we started to adopt Apollo Federation into our stack. One gateway service lives in front of our other GraphQL services (which have their own databases). This gateway is what the client-side apps ("Web Browser" in diagram) queries to get its data. For clients to start using the gateway API every dependent service must be up and healthy.

My goal is to start the entire set of services in a deterministic manner, have each service wait until its dependent service is running and healthy. I'll focus on the database and migrations in this post but the concepts here extend to any service having dependencies on another service.

A typical Docker Compose file which runs the products service with it's migrations will look like this:

version: "3.7"

postgres:
  image: yourorg/postgres
  command: postgres -c 'max_connections=1024'
  expose:
    - "5432"
  environment:
    POSTGRES_USER: ${DATABASE_USERNAME}
    POSTGRES_PASSWORD: ${DATABASE_PASSWORD}
  volumes:
    - ${YOUR_ORG_INSTALL_PATH}/volumes/services/postgres-data:/var/lib/postgresql/data:delegated

products-run-migrations:
  build:
    context: ./apps/products/migrations/.
  image: yourorg/products-run-migrations:${IMAGE_TAG:-latest}
  env_file:
    - ./apps/products/products.env
  depends_on:
    - postgres

products:
  build:
    context: ./apps/products/.
  image: yourorg/products:${IMAGE_TAG:-latest}
  env_file:
    - ./apps/products/products.env
  depends_on:
    - postgres
    - products-run-migrations

A few issues with this setup, focusing on depends_on:

1. The products-run-migrations script will run right when postgres starts to run, not when it's healthy and ready to accept connections. There is a 99% chance that the postgres container will not be healthy when the migrations begin to run, causing them to fail.
2. Similar situation with the products service, it will run right when products-run-migrations starts to run. If you query the products service when it's healthy, there is a good chance the migrations have not yet completed.

What We Want

This is currently not what we want, ideally developers should be able to start the products service and when it's healthy, know that the migrations ran successfully and they can query its API.

This required me to do some research.

I found out there is a conditions argument to depends_on I could use to achieve the implementation we wanted. But the next thing I found out is where things went haywire.

We needed to downgrade our docker-compose version.

Turns out, Docker Compose 3.x versions are meant to be used for Docker Swarm and Kubernetes environments where services are not strictly dependent on each other. This promotes a more fault-tolerant environment where services run independently.

What I saw people suggesting was to switch to Docker Compose 2.4 and use a port waiting script like wait-for-it.

Our new Docker Compose 2.4 file:

version: '2.4'

postgres:
  image: yourorg/postgres
  command: postgres -c 'max_connections=1024'
  expose:
    - '5432'
  environment:
    POSTGRES_USER: ${DATABASE_USERNAME}
    POSTGRES_PASSWORD: ${DATABASE_PASSWORD}
  volumes:
    - ${YOUR_ORG_INSTALL_PATH}/volumes/services/postgres-data:/var/lib/postgresql/data:delegated
  healthcheck:
    test: ['CMD-SHELL', 'pg_isready -U root']
    interval: 60s

products-run-migrations:
  build:
    context: ./apps/products/migrations/.
  image: yourorg/products-run-migrations:${IMAGE_TAG:-latest}
  entrypoint:
      - /bin/bash
      - -c
      - 'wait-for-it $$DATABASE_HOSTNAME:5432 -s -t 60 -- npm run db-migrate:products
  restart: on-failure:5
  env_file:
    - ./apps/products/products.env
  depends_on:
      postgres:
        condition: service_healthy

products:
  build:
    context: ./apps/products/.
  image: yourorg/products:${IMAGE_TAG:-latest}
  env_file:
    - ./apps/products/products.env
  restart: on-failure:5
  depends_on:
      postgres:
        condition: service_healthy
      products-run-migrations:
        condition: service_started

A few new additions:

1. Added a healthcheck to the postgres container. This made it so migration scripts like products-run-migrations can start to run only when the database is ready to accept connections.
2. We added an entrypoint along with the condition arg to depends_on to our migration images. This made sure that not only the database is running but that the port is returning a 200 response code.
3. Added condition args to both depends_on in the products service definition.

It's not perfect but works much better than before. The issue still remains of the app not waiting for the migrations to be fully finished before booting up. If they run relatively quickly, local developers may not run into problems. In CI, we have full control over the environment so we can add an extra command to skirt around this issue.

# .circleci/config.yml
test-products:
  executor: node-docker
  steps:
    - test-project:
        project-name: products
        pre-test-command: docker-compose run -T products-run-migrations
        tests:
          - run-project-test:
              project-name: products

The Issues That Still Remain

What I didn't touch on is what our production environment looks like in terms of Docker Compose. We decided to maintain two versions of our Docker Compose files, one in 2.4 and one in 3.7. This is because we want to adopt Kubernetes easily in the future. You may stick with one or the other but for a great local development experience, we decided to always have a 2.4 file going.

With so many choices for which technology to use when building a website, it can get overwhelming. You need to consider who is going to use it, what content to display and who will maintain it. A static website is a great choice when creating a blog, band website or e-commerce store. Static websites are an ode to the past when websites were just plain-ol files on a server you accessed via URL. They provide benefits like being fast, having great SEO and not being dependent on a certain runtime like PHP. This is in comparison to a server-rendered website like what you would have with Wordpress, Drupal or Ruby on Rails.

Static websites are built using static assets. The next question becomes where to store (and manage) this content. If you are a solo webmaster, the content can be files in a Git repo. If you have clients or other developers who will want to manage the content, a CMS (Content Management System) is what you need. A CMS is a service which stores your website content, for example blog posts and concert dates.

Cosmic CMS!

With Next.js SSG, we are using the CMS in a "headless" fashion. After trying a bunch of Headless CMS offerings, one I've stuck with is Cosmic. Cosmic is an intuitive, powerful, and simple-to-use service which lets us get up and running quickly. They provide many starter apps that you can preview or fork. For example, I chose the Next.js Static Blog and had a production version of the website running in under 5 minutes.

Choosing the Tech

Next.js with GraphQL is my personal choice when it comes to Static site development. Next.js is a hybrid React framework which supports building static websites. It also lets you build server-side rendered pages when the need arises. It handles routing, has a large community supporting it making it one of the best ways to build a React app in 2020. The other tech you may have heard also does this is Gatsby.js. Gatsby is more user-friendly but is more opinionated with its technology choices (forced use of GraphQL versus it being a choice).

We are choosing to use GraphQL over the Cosmic NPM module. GraphQL is a standardized way to get data from services and is a great choice when needing to get data from a CMS. As you create custom data types in Cosmic, you are able to query for it in the GraphQL API. One of the benefits of using GraphQL is that you are less dependent on a specific SDK.

Tutorial

For reference, I forked the example Cosmic Next.js project here.

Creating the Cosmic Project

After creating an account on Cosmic and going through the product tour, you will be shown the “Create New Bucket” screen.

Click "Start with an App" then search and select "Next.js Static Blog" from the list of apps presented. This will do many of things.

1. Create a Cosmic bucket
2. Create sane data-types inside the bucket for use with a blog
3. Fill the bucket with demo content

Here is what the created bucket looks like on your Cosmic dashboard

Next.js local development

The next thing we need to do is clone the Next.js code to our local environments. This will enable us to run the Next.js locally and pull content from the demo Cosmic bucket we created.

git clone git@github.com:aleccool213/nextjs-cosmic-graphql-app.git

You can also choose to create a GitHub repository for yourself using the template.

Once inside the new directory, make sure you are using the correct Node.js version by using NVM.

nvm use v12.18.3

Install Yarn and install the project dependencies.

brew install yarn && yarn

Run the app!

yarn dev

Almost there!

Cosmic Environment Variables

Before we are able to query the Cosmic GraphQL API, our app needs to know where it lives. Environment Variables are deployment specific values which contain sensitive things like API keys.

There are three env vars we need to define to have the app work locally. Create a file named .env.local (don't worry it's ignored by Git), it should look like this:

COSMIC_BUCKET_SLUG=demo-nextjs-static-blog
COSMIC_READ_KEY=77H1zN7bTktdsgekxyB9FTpOrlVNE3KUP0UTptn5EqA7T0J8Qt
# Preview secret can be anything you choose
COSMIC_PREVIEW_SECRET=iwvrzpakhaavqbihwlrv

To get these values, head over to the Settings sidebar menu in your bucket, and click "Basic Settings".

Run the app again with yarn dev

And we are up!

Looking inside the box

There are two things that we need to understand when it comes to Statically-Generated Next.js apps, pages and routes. Pages are content which depend on external data, and routes are URL routes which depend on external data. Both have you defining special Next.js specific functions, getStaticProps and getStaticPaths.

The file which contains the logic for generating page content based on the Cosmic GraphQL API is located at pages/posts/[slug].js.

export async function getStaticProps({ params, preview = null }) {
  // Get the data from the API
  const data = await getPostAndMorePosts(params.slug, preview);
  // Convert markdown content to HTML content
  const content = await markdownToHtml(data.post?.metadata?.content || "");
  return {
    props: {
      preview,
      post: {
        ...data.post,
        content,
      },
      morePosts: data.morePosts || [],
    },
  };
}

export async function getPostAndMorePosts(slug, preview) {
  // Query for the data through the Cosmic GraphQL API using Apollo Client
  ...
  const moreObjectsResults = await client.query({
    query: gql`
      query getPostQuery(
        $bucketSlug: String!
        $readKey: String!
        $status: status
      ) {
        getObjects(
          bucket_slug: $bucketSlug
          input: {
            read_key: $readKey
            type: "posts"
            status: $status
            limit: 3
          }
        ) {
          objects {
            _id
            slug
            title
            metadata
            created_at
          }
        }
      }
    `,
    variables: {
      bucketSlug: BUCKET_SLUG,
      readKey: READ_KEY,
      status,
    },
  });

This is one example of a page using getStaticProps. It is also used in the Index page for getting all the blog post titles and excerpts.

pages/posts/[slug].js also contains getStaticPaths which tells our Next.js app which routes to generate.

export async function getStaticPaths() {
  // Get all post data (including content)
  const allPosts = (await getAllPostsWithSlug()) || [];
  return {
    // Tell Next.js all of the potential URL routes based on slugs
    paths: allPosts.map((post) => `/posts/${post.slug}`),
    fallback: true,
  };
}

After understanding these two aspects, the blog is just a regular React app.

Deploying

Now that we have our website working locally, let's deploy it to Vercel. The first step is making sure you have the code in a Git repository.

I recommend you have the code in GitHub. You can use the GitHub CLI to create a repo in your current directory with gh repo create.

We now need to sign up for Vercel and have it use the code from the GitHub repo. You can sign up for Vercel with your GitHub account here. You can import the code from GitHub using the "Import Project" feature.

When importing the project, make sure you define the environment variables, COSMIC_BUCKET_SLUG, COSMIC_READ_KEY, and COSMIC_PREVIEW_SECRET.

When deployed, all pushes to your default Git branch will have Vercel deploy a new version of your website!

Bonus

Previewing

The Next.js docs on preview mode are right here.

Local development and deploying the website to production will cover most of your use-cases. Another common workflow is saving a draft of changes on your CMS and then previewing those changes on your local machine. To do so, we will enable "Preview" mode both on Cosmic and our Next.js app.

First thing we will need to do is have Cosmic know that the Posts object type will be preview-able. On the Posts setting page, add the preview link.

http://localhost:3000/api/preview?secret=iwvrzpakhaavqbihwlrv&slug=[object_slug]

When finished, click "Save Object Type".

Let's try editing a post and see it show up on our local machine. Try changing the title of "Learn How to Pre-render Pages Using Static Generation with Next.js" and click "Save Draft" instead of "Publish".

The Save Draft button

We now have unpublished changes. Run the app locally with yarn dev and then click "Preview" right under "Save Draft".

Our preview mode!

Webhooks

Note this feature requires a Cosmic paid plan

The only way to deploy new content to our blog is to have a developer push to the default git branch. This action will trigger Vercel take the new code and push a new version of our website. We ideally want our content creators to have the same control. Webhooks are a way we can do this.

Let's set up a webhook which lets Vercel know when our posts in Cosmic have new updates. This will let us deploy new versions of the website without developers needing to do anything.

Go to the Git integration settings page (https://vercel.com/[project]/settings/git-integration) in your Vercel project and create a new Deploy Hook named "Cosmic Hook".

What your settings should look like when the webhook is created

Now over in Cosmic settings, we can add this webhook. Let's add Cosmic notify Vercel when changes get published. You can see how we can do this for previews as well if we wanted to in the future.

Edited/Created and Published!

To test this go to the same post we tested Previews with and add some content to the end of the article and publish. You should see a deploy happen on Vercel with the new content deployed to the live version of your website!

Conclusion

Want to see what the final website looks like? Click here to check it out.

Like this post? Subscribe to my email newsletter for more like it in the future.

HEY targets those who are tired with existing solutions, the users who see email as a chore and not what it should be, a delightful, curated experience. HEY gives you complete control over where email should go, who is allowed to give you a push notification and so on. Shifting away from automated filters and into a more controlling experience is tough at first but is worth it in the end.

I switched from Gmail to Fastmail years ago for simplicity and enhanced privacy. There is something to be said about spending money on a small business versus one which cares about its ad revenue more than it's users that lures me to a service. HEY is a big step forward in the same direction as Fastmail.

Email Hygiene

Basecamp is looking at email from the ground up. Jason Fried, founder and CEO of Basecamp, says it well in his walk-through of the product:

One of the problems with email is that everybody can email you, which is also one of the great things about email

I see myself as a hygienic email user, using filters and labels as much as I can, and try my best to combat the fact that anyone has access to my inbox. A few custom rules to put emails to certain folders, rules to make my inbox feel a more clear and less cluttered. For example, I have a newsletter rule where emails I don't care about go (sorry to the marketers who are reading this) and a rule for receipts. This happened to map 1-1 with HEY's "The Feed" and "The Paper Trail". This meant that using HEY was already familiar to me. It made my workflow for creating rules much easier with "The Screener" which buckets sender's into these categories with unmatched speed. I can say that these buckets were enough. Non-categorized senders turned out to be important and kept going into my "Imbox". If they abused the fact they had this access, I would screen them out, banished to the shadow-realm, my spam folder.

When all the filters are at full steam, checking email is a calming and less stressful experience. This is something I never thought email would make me feel. Another reason for this is that no notifications are sent to you for new emails by default, you need to choose who is able to notify you. This falls in line with how much control HEY gives you, it lets you decide what is important.

The Imbox

Ultimate privacy

Okay, get this, every email that you receive has a tracker inside it, it can tell when you opened it, where you opened it and how many times you opened it. Well, not every single one, but every email you get might have it because you wouldn't be able to tell the difference. Senders put a 1x1 HTML image element inside an email (hidden from view), when you open it this image gets loaded from the sender's server. This is how a bunch of information like I described earlier is then sent back to the email sender. Blocking this behaviour is as easy as not loading images when you open an email, which no email client does by default, it needs to be set by the user. The other problem is that if you turn that on, you can hardly read emails because usually, the content depends on images loading. HEY brings a feature that surprises me no one else is doing yet, filtering out those pixel trackers.

Filtering out pixel trackers is akin to a DNS server that has ad-blocking enabled. I already use one of these called NextDNS. It works well and makes it so I don't have to do ad-blocking client-side, for example with a browser extension. How HEY does this is that it loads images within your email on a server within their network. This makes the sender think the email has been opened by you (but it was opened by HEY). HEY then shows you these pre-loaded images from their server, along with the email it was placed in. Being the privacy-oriented person I am the price tag for HEY may be worth it just for this.

The horror, vendor lock-in

To do this magic, Basecamp decided to not let users integrate using email protocols such as IMAP, POP3 and SMTP. This means you can't use HEY with any other client than their own. Mail for MacOS, Thunderbird, yup, all out of the question. This is a con of the service, other email clients have been in development for eons and come with useful features, too many to mention here. HEY comes with native apps for every device you use but it stills feels wrong to not talk about it. I get it why they don't want to support it, most of HEY's features wouldn't make sense on a traditional email client. For example, screening senders, merging/renaming threads, and others.

If you decide to switch off of HEY, exporting all your emails is possible so you don't have to worry about your history being lost. HEY promises to forward your emails sent to your old HEY address to a new address if you do decide to switch (that is if you paid for HEY in the first place).

Yet another thing to "learn"

The problem with new software products is that the learning curve usually is steep. New product fatigue is hard to deal with and this is why new products that come out try to steal features from existing solutions for easy adoption. HEY suffers from this, you will need to learn how to use it or you will get lost and won't feel comfortable switching. An on-boarding experience and tutorials covering HEY's features help but I felt like these need improvement. I felt sad when the tutorials stopped coming, I wasn't comfortable to navigate my email alone yet. Nuances like moving emails around to different folders and not knowing if future emails will go there. Little things to criticize but shouldn't stop anyone from enjoying what Basecamp has built.

Conclusion

Getting to this point took some strong-willed people willing to take a risk, going up against established players in the game. HEY isn't a perfect email app but it pushes the boundaries of how email should work. This alone is commendable, most companies don't have the gusto to do this. I decided not to pay for a full-year sub as they don't support custom domains and switching email addresses was painful the last time I did. I only have the energy, time and motivation to do this once a decade. They have a form to stay up to date and get notified when this feature ships as they know it's the reason a lot of people are waiting to pay.

Show Time

Let's say we have a component which renders a list of Github issues showing their title. In the Container Component pattern, we would have a "container" component, GithubIssueListContainer, which handles running the query. After this, it passes down the data to its presentational components which need it to render, GithubIssueInfoCard.

const GITHUB_ISSUES_LIST_QUERY = gql`
  query GithubIssuesListContainerQuery {
    organization {
      id
      name
    }
    issues {
    totalCount
    pageInfo {
      endCursor
      hasNextPage
    }
    edges {
      node {
        id
        title
        description
      }
    }
  }
`;

const GithubIssueListContainer = () => {
  const { loading, error, data } = useQuery(GITHUB_ISSUES_LIST_QUERY);
  return (
    {data.issues.edges.map(
      edge =>
      (
        <span key={edge.node.id}>
          <GithubIssueInfoCard issueDetails={edge.node} />
        </span>
      ),
    )}
  );
}

interface GithubIssueInfoCardProps {
  issueDetails: {
    id: string;
    title: string;
    description: string;
  }
}

const GithubIssueInfoCard = ({ issueDetails }) => {
  return (
    <>
      {issueDetails.id} {issueDetails.title} {issueDetails.description}
    </>
  )
}

The issue here is that GithubIssueInfoCard is dependent on its parent component in its knowledge of where data comes from in the GraphQL graph.

If we want to render a new field from the graph, e.g. labels, we will need to add that to the query in GithubIssueListContainer and pass that down to GithubIssueInfoCard via props. This requires changes to the both the query in GithubIssueListContainer and the props in GithubIssueInfoCard.

This is the Way

Following along our mantra of isolation, how about if GithubIssueInfoCard defined what data it needs to render from the GraphQL graph. That way, when we make changes to what data this component, only this component needs to change.

const GITHUB_ISSUES_LIST_QUERY = gql`
  ${GITHUB_ISSUE_INFO_CARD_FRAGMENT}
  query GithubIssuesListContainerQuery {
    organization {
      id
      name
    }
    issues {
      totalCount
      pageInfo {
        endCursor
        hasNextPage
      }
      edges {
        node {
          ...GithubIssueInfoCardFragment
        }
      }
    }
  }
`;

const GithubIssueListContainer = () => {
  const { data } = useQuery(GITHUB_ISSUES_LIST_QUERY);
  return (
    {data.issues.edges.map(
      edge =>
      (
        <span key={edge.node.id}>
          <GithubIssueInfoCard issueDetails={edge.node} />
        </span>
      ),
    )}
  );
}

export const GITHUB_ISSUE_INFO_CARD_FRAGMENT = gql`
  fragment GithubIssueInfoCardFragment on Issue {
    id
    title
    description
  }
`;

interface GithubIssueInfoCardProps {
  issueDetails: {
    id: string;
    title: string;
    description: string;
  }
}

const GithubIssueInfoCard = ({ issueDetails }) => {
  return (
    <>
      {issueDetails.id} {issueDetails.title} {issueDetails.description}
    </>
  )
}

This might seem odd at first, but the benefits are worth it. As with anything in programming it doesn't come without tradeoffs.

Benefits

Less parent component coupling

When components define the data it needs to render, it de-couples the component from its parent. If for example you wanted to show GithubIssueInfoCard on another page, import the fragment into that container component to get the right data fetched. e.g.

import {
  GITHUB_ISSUE_INFO_CARD_FRAGMENT,
  GithubIssueInfoCard,
} from "./GithubIssueInfoCard";

const NOTIFICATIONS_LIST_QUERY = gql`
  ${GITHUB_ISSUE_INFO_CARD_FRAGMENT}
  query NotificationsContainerQuery {
    notifications {
      totalCount
      pageInfo {
        endCursor
        hasNextPage
      }
      edges {
        node {
          id
          eventText
          eventAssignee {
            id
            avatar
            username
          }
          relatedIssue {
            ...GithubIssueInfoCardFragment
          }
        }
      }
    }
  }
`;

Types become easier to maintain

If using a TypeScript, you likely are generating types from your GraphQL queries. A large benefit of our new pattern comes with defining props in components. You can define the data it needs to render as a type from our generated type file.

import { GithubIssueInfoCardFragment } from "../../graphql-types";

interface GithubIssueInfoCardProps {
  issueDetails: GithubIssueInfoCardFragment;
}

When the fragment changes, after you generate types, no prop changes needed!

Less chance of changes when developing component first

With Storybook becoming popular, many developers are starting to develop components in Storybook first and the integrating them into the app at a later time. What may happen is that in app integration, props are defined incorrectly.

Defining the fragment of the GraphQL graph this component needs to render, there are less chances of code changes when integration happens due to forcing the developer to know the exact shape of the data it needs to render. This of course is only possible defining the api in advance which sometimes isn't always the case.

Trade-offs

Of course, like everything in programming, there are trade-offs in this approach. It's up to you to see if it's worth it.

Presentational components are not generic

The crummy thing is that our presentational components become more coupled to the application and API data model. If we want to migrate over to a component library for others to use, these components will need to be refactored to have their fragments removed. It's not too much work, but it is more work than the alternative.

Fragments sometimes become difficult to manage

Importing many fragments into a single GraphQL query isn't the best experience. If we have many presentational components within a container component, importing them all can be hairy. Sometimes you may forget to import the fragment and Apollo can return some unhelpful messages.

const GITHUB_ISSUES_LIST_QUERY = gql`
  ${GITHUB_ORG_INFO_CARD_FRAGMENT}
  ${GITHUB_ISSUE_COUNT_CARD_FRAGMENT}
  ${GITHUB_ISSUE_INFO_CARD_FRAGMENT}
  query GithubIssuesListContainerQuery {
    ...GithubOrgInfoCardFragment
    issues {
      ...GithubIssueCountCardFragment
      pageInfo {
        endCursor
        hasNextPage
      }
      edges {
        node {
          ...GithubIssueInfoCardFragment
        }
      }
    }
  }
`;

Conclusion

We have been using this pattern at Yolk for a while now and it has grown on everyone. We develop our components first in Storybook and it forces the developer to understand where the data is coming from and ask questions about the data model and it's usage.

Maintaining an open-source package can be a time-consuming task. Issues to be triaged, pull requests to be reviewed and changelogs to write. Publishing new versions of the code is usually done manually and making it automated is often on the back-burner of the maintainers' to-do list. There are a couple of key features of a rock-solid release process, the changelog, Git tags, NPM versions, and enforcing Semantic Versioning. Keeping all these in sync makes it so users understand changes in a release and understand how to keep up-to-date. Maintainers who fail to perform all of these steps will have a hard time triaging issues, which leads to more time debugging and less time spent coding. I recently came across a combo of tools, semantic-release and Github Actions, which made the entire release process automated, transparent, and simple to understand.

How It Works

Before we talk about implementation, it's important to understand what work our tools will perform. That way, if there are problems or modifications, we can fix them. semantic-release is going to do the majority of the work here, they say it best on their README.

It automates the whole package release workflow including determining the next version number, generating the release notes and publishing the package.

The Next Version Number

During a release, to determine the next version number, the tool reads commits since the last release. It knows your last release by looking at your Git tags. Based on the type of commit, it can determine how to bump up the version of the package. For this to work, commits need to be written in a certain way. By default, semantic-release uses the Angular Commit Message Conventions. This is critical because consumers of the package need to know if a new version releases a new feature, introduces breaking changes or both. For example, if someone commits fix(pencil): stop graphite breaking when too much pressure applied, semantic-release knows this contains a fix and to create a patch release. This will increase the version in the minor version range (0.0.x).

Never seen this type of versioning before? Check out Semantic Versioning.

After analyzing all the commits, it takes the highest priority type of change and makes sure that is the one that is applied. For example, if two commits were introduced since the last release, one breaking (x.0.0) and one minor (0.0.x), it would know to just up the version by breaking range.

Generating Release Notes

Once it has done finding out what type of release the next version is, changelog notes are generated based on the commits. semantic-release doesn't use conventional CHANGELOG.md file to notify users of what has changed, it does so with a Github Release which is attached to a Git tag.

An example of a Github Release that semantic-release generates and pushes on builds.

Automating With Github Actions

So semantic-release will be the tool to perform most of the work, but we still need a service to run the tool on. That is where Github Actions comes into play. When pull-requests are merged into master (or any base branch you configure), Github Actions will run a job that simply runs semantic-release with your configuration. All of the work we described previously will be performed.

An example of a Github Actions run using semantic-release to publish a new release.

Steps to Take

We will be using as many defaults as possible to make configuration dead simple. semantic-release uses a plugins system to enhance functionality. Here are the default plugins semantic-release uses.

Let's go over the steps which will make this all run smoothly.

1. Add a dummy version property to the package.json of package. Released code will have the proper version written to this file by semantic-release.

        "version": "0.0.0-development",

1. Add a new property to the package.json, publishConfig. This will be the home of our semantic-release configuration.

        "publishConfig": { "access": "public", "branches": ['master'] }

1. The last step is to create a Github Action YAML file. This will tell Github Actions what to do when a commit is made to the repository.

        # .github/workflows/test-and-release.yaml

        name: Test and Release
        on: [push]

        jobs:
        test-and-release:
            name: Run tests and release
            runs-on: ubuntu-18.04
            steps:
            - name: Checkout
                uses: actions/checkout@v1
            - name: Setup Node.js
                uses: actions/setup-node@v1
                with:
                node-version: 12
            - name: Install dependencies
                run: npm ci
            - name: Run tests
                run: npm test
            - name: Release
                env:
                GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
                NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
                run: npm run semantic-release

1. Add NPM_TOKEN to the secrets in the Github repos settings page. You can generate one of these from your NPM account at https://www.npmjs.com/settings//tokens

   

And that's it! You have a fully automated package release process 🎉

Bonus

I implemented this on a repo we recently open-sourced at Yolk AI. It's named next-utils and everything described here can be found there.

{% github Yolk-HQ/next-utils %}

Another great thing about using semantic-release with Github Actions is that it has out-of-the-box support for bot comments. It will go into every issue and pull-request closed since the last release and comment to make sure everyone is aware. Here is an example:

{% github https://github.com/Yolk-HQ/next-utils/issues/12#issuecomment-581484992 %}

If you liked this post, check out more at https://blog.alec.coffee and signup for my newsletter

I set out to search for a new tracking tool for my blog. My criteria for choosing a new product were:

• Be simple
• Have features I will use
• Put a focus on privacy
• Built with a programming language I know so making changes is easy
• Be able to easily host on a Platform-as-a-Service like Heroku
• Have the ability to be easily added to a Gatsby blog
• Have an option to not collect unique user data such as OS, Browser Info, Device & ScreenSize

Meet Ackee

Beautiful, isn't it

I came across Ackee 🔮, a self-hosted analytics tool. This tool fit my requirements almost perfectly. It is built using Node.js which I have experience in and it focuses on anonymizing data that it collects. More information on how Ackee anonymizes data here.

The steps you need to take to start collecting statistics with Ackee are to start running it on a server, Heroku in my case, add the Javascript tracker to your Gatsby site and test to see if the data is flowing correctly.

This a detailed guide on how I went about deploying it to Heroku. Afterwards, I contributed back a Deploy-to-Heroku button which deploys it in one-click. Find the button here.

Up and running on Heroku

First thing is to start running the server which is going to receive the tracking data from your website.

1. Create a new Heroku app instance

   

2. Use the heroku-cli to upload the code

   # clone the code
   git clone git@github.com:electerious/Ackee.git
   
   # login to heroku
   heroku login
   
   # add the heroku remote
   heroku git:remote -a ackee-server
   
   # push the code
   git push heroku master
   

3. Configure a MongoDB add-on, this is where the data will be stored

   

4. Configure the environment variables

   heroku config:set ACKEE_PASSWORD=<your password>
   heroku config:set ACKEE_USERNAME=<your username>
   

And voila! You are finished, that was easy, wasn't it? Open the webpage Heroku automatically configures for you, it should be https://ackee-server.herokuapp.com/, you should see this:

The log in page!

Adding the tracker

Now we need to send data over from the website to the server we now have running on Heroku. If you are using Gatsby, this is incredibly easy with the plugin.

1. Install the tracker

   npm install gatsby-plugin-ackee-tracker
   

2. Create a domain on Ackee and get the domain id. Find this option in the settings tab of your Ackee instance.

3. Add it to your Gatsby config

{
    resolve: "gatsby-plugin-ackee-tracker",
    options: {
        // Domain ID found when adding a domain in the admin panel.
        domainId: "<your domain id>",
        // URL to Server eg: "https://analytics.test.com".
        server: "https://ackee-server.herokuapp.com",
        // Disabled analytic tracking when running locally
        // IMPORTANT: Set this back to false when you are done testing
        ignoreLocalhost: true,
        // If enabled it will collect info on OS, BrowserInfo, Device  & ScreenSize
        // False due to detailed information being personalized:
        // https://github.com/electerious/Ackee/blob/master/docs/Anonymization.md#personal-data
        detailed: false
    }
},

1. Run the site locally

   gatsby develop
   

Testing to make sure it worked

Open up your site at http://localhost:8000 and go to a new url.

Observe the network requests your site is sending. You will notice it now sends requests to your Heroku instance.

Using the dev tools

And with that, we now have the server running Ackee and our Gatsby sending analytics!

What you get

Let’s explore Ackee, shall we.

Home page with total site views

List of referrers

Per page view count

Alternatives

Here are some alternative methods I considered when thinking about analytics for my blog.

No tracking

Combined with the fact more and more people are blocking trackers all-together (Firefox, Brave and Chrome ad blocking extensions), JavaScript-based tracking is becoming less and less valuable over-time. Most analytics can easily become a way to be vain about your blog and you can start a bad habit of always checking them (wasted time compared to producing actual content). Deciding not to track any analytics at all is not a bad decision these days.

Server-side analytics

The most private and fast way of collecting analytics on your website may be to collect analytics at the server level. What this means is instead of using a JavaScript tracker (which may be blocked by the browser), stats are collected when the HTML is sent from the server. Integration with your static host provider or DNS provider is needed here. The main con about this method is that data is collected by a third party service and also is usually not free. Cloudflare offers these types of analytics alongside Netlify. A huge benefit is the ease of setup, usually the provider just turns it on with a switch on their side, no setup needed from you.

In the past, web-app end-to-end testing has been a tricky beast. Selenium has been the main solution for quite some time and has a huge history. It has great browser compatibility but having your tests be consistent is difficult because it wasn't designed for app testing. That's why I got so excited when I heard about Cypress, promising to fix all of the old and broken ways of past frameworks. After writing and reviewing close to 200 test scenarios in the past year (that's with a small team), I wanted to write about what I wish I knew when I started and share my thoughts on my journey with Cypress thus far.

What's in the box

End-to-end testing has always been a fragmented experience. You need to bring a lot of your own tools, for example, a test runner, an assertion library, and maybe other things like mocks. With Cypress, it packages all of those things together, this makes set up and configuration, dead simple. Not only that, the documentation is some of the best I have ever read in my career, with guides on everything you are likely to encounter. Not only do they do a great job telling you how to use the product, but have in-depth explanations on the architecture, flakey tests and best practices.

Prototyping

{% vimeo 379033499 %}

If you have the chance, before adopting anything of this scale, I always think it's a good idea to test it on a small project first, just to get a feel. Before advocating for it, I added it to my personal blog, just to see how the experience was.

A very simple scenario:

• Load up the app
• Go to the index page
• Click the first blog post link
• Assert content shows up

I was blown away with how fast it took me, under an hour. This was really as simple as writing a few lines of Javascript for the test itself, the npm script in the package.json, and running it in CircleCI. Not only did Cypress perform the assertions but also it was recording videos! This could have been an even faster setup if I used the CircleCi Cypress Orb.

This got me a huge amount of test coverage in very little time. This proof of concept was more than enough to convince my team that Cypress was the right choice when it came time to start writing end-to-end automated tests.

Decisions and Tradeoffs

The browser-based products we have at Yolk are completely separated from the server-side API's they fetch data from, they build and are served separately. This presents a few ways forward when deciding to write end-to-end tests. You can either deploy your backend with your frontend and test as if the app is in production or completely mock out API responses. Using a real backend means spinning up potentially memory-intensive processes when running on CI but you get the assurance that apps are near-production. With mocking your API responses, you test less of your stack, risk stubbing out non-realistic responses, and have to worry about the extra maintenance of keeping them up-to-date.

We decided on deploying live instances of the backends related to the app we were testing. This decision was easy for us to make due to already having a CLI tool to do much of the hard work. This tool (aptly named yolk-cli) downloads the latest docker images for apps and knows how to spin up products with minimal configuration. This made getting the real APIs working on CI not too huge of a task.

Turns out, running two or three large python apps and a few Next.js servers on CircleCI does crap out the memory limit pretty fast. We reached out to CircleCI and they gave us access to their large resource classes (up to 16gb of RAM), score!

Seeding Data

The next challenge we faced was seeding data. Your test scenarios must share as little state as possible with each other. This is a testing fundamental and Cypress addresses it in their guides. Having test scenarios data-independent goes a long way when debugging why things are going wrong. On the flip side, having all of your data be created through the UI will make for slow tests, there is a balance. This will be highly customized to how your app works but I will go into what worked for us.

Going back to our cli tool once again, it had a few commands which seeded some basic data. The commands looked like this:

yolk seed-articles

yolk seed-bots

Getting your feet off the ground with data that is basic to your app, static data or very high-level entities, for example, will speed up this process and will be easy to run on each CI build.

The next part will be seeding data for entities that may be more specific to individual tests. This is where things get contested, there is no silver bullet for this. We decided to call the APIs directly for these situations and use Cypress custom commands to initiate these requests. This was a decent choice because we are using GraphQL; the custom commands that use the API were easy to write and document.

{% gist https://gist.github.com/aleccool213/5df73bab0eff93f4be583178e32a6554 %}

Writing custom commands for actions which your tests will be performing over and over are a great way to consolidate all code, not just data seeders!

Writing Scenarios with Gherkin

If you have written end-to-end tests before, you may be familiar with Gherkin syntax, used by Cucumber. This is an expressive, English-like way to write test scenarios. It can help with documenting your features and non-developers can contribute to writing test cases. We found a way to integrate this file syntax into Cypress using a plugin.

{% gist https://gist.github.com/aleccool213/2747de9793028d16c8cad3e3f6fb3b85 %}

After writing these commands, the plugin will then go to Cypress to actually run the implementations:

{% gist https://gist.github.com/aleccool213/8b5af73f64791b6d551787e06bab05d5 %}

Asserting Elements and Best Practices

When it comes down to it, end-to-end testing is just making sure elements on the page have the correct content. When writing Cypress tests, 90% of the time you will need to be selecting elements and peering inside them. Cypress has a standard get() command which exposes a JQuery-like selector to you, this should be familiar to those who have worked with Selenium. The problem with this selector is that it can be used incorrectly and you can't enforce (with code) it's usage. Welcome cypress-testing-library, a wonderful tool maintained by a great testing advocate in the community, Kent C. Dodds.

This plugin exposes a myriad of commands prefixed with find which work similarly to how get() does in native Cypress. All of these commands make for selectors that are resilient to change. This can have a dramatic effect on how your tests stay consistent as your application progresses.

Debugging

If you have ever worked with Selenium before, you know that debugging end-to-end tests can be somewhat of a nightmare. With Cypress, this pain is at an all-time low. Being a focus of the core product, being able to debug is one of the more pleasant experiences in your Cypress journey. Like for most things, they have a great guide to get you started.

Most of the things they have mentioned are great but the case which you will likely run into the most is a selector being incorrect. For this type of scenario, the GUI is a great way to find out what is going wrong. There is a nice video explaining how to write your first test and it shows the GUI in action.

Visual Testing and Catching Regressions

Another critical part of end-to-end testing will be how things look. HTML and CSS play a huge part in how your application will look like in different scenarios. Cypress can give you a lot of coverage in terms of how your app works but starts to break down when you want to assert its looks. Especially when it comes to browser compatibility and the different screen sizes your application is used in, visual regressions are hard to catch without proper implementation of Visual Snapshot Testing.

The solution we ended up with was Percy as it integrates nicely with Cypress and Storybook. What it can do is take the current HTML and CSS which is being rendered in your Cypress test scenario and send it over to Percy's servers. Percy then renders the markup on its own internal browsers, with Chrome and Firefox being options. Percy knows which feature branch your Cypress test is being run in and compares this with your configured base branch. This can give you great confidence in pull requests when you don't know if code is changing the look of a certain component in your application. This can be a big time-saver if you have a lot of code in your Cypress tests that assert css values or how things should look.

Hot Tip: You can have Cypress take snapshots locally and then with Percy only when it's enabled by creating a new takeSnapshot custom command:

{% gist https://gist.github.com/aleccool213/e0445d07bdab874cea06eb96284d2661 %}

Parallel Builds and the Cypress Dashboard

Once test runs start to become long enough, you will start looking for other strategies to speed them up. Parallelization is something that can be performed due to Cypress running feature scenario files with a clean state each time they are run. You can decide on your own balance strategy, how your tests can be broken up, but the hosted version of Cypress Dashboard provides a way to do this automatically.

Let's say I can afford to have three CircleCI containers to run my Cypress tests. First, I define the parallelism: 3 in my CircleCI job step config. What this will do is spin up three instances of your job, all with different job ids. Pass those ids off to Cypress, and you are off to the races. If you have Cypress Dashboard set up correctly, that service will tell your container which tests it should run. Here is an example of the config:

{% gist https://gist.github.com/aleccool213/6e7ba6f9f3d975509a207e3c6c95923f %}

The super neat thing about this is that Cypress Dashboard knows your past test history and their speeds. It will use this knowledge to optimize your parallel builds by making sure the containers get a balanced load of tests to run!

Don't worry if this doesn't make much sense to you, Cypress has answered how to do this.

Browser Support

Unfortunately, if your organization needs to have support for IE11, you are out of luck. The Cypress team has explicitly said they won't be supporting it. There is an incredible thread on Github that I really hope you read through. It goes into why they are rolling this out slowly and didn't choose WebDriver from the beginning and wrote their own custom driver.

For us at Yolk, we needed IE11 support for a couple of our applications. We kept getting regressions within IE11 and needed more comprehensive test coverage. We decided to use Browserstack Automate and Selenium to cover these apps. For CI, we already had the app built and running in Cypress, we just needed to add a new build step that ran these tests using the Browserstack Local Proxy.

For the tests themselves, we decided to integrate Selenium with Cucumber, a common pairing. To make this process easier, we copied our Gherkin .feature files over to a new folder and wrote specific Selenium-based step implementations.

A cool concept I had an idea for was to re-use the same .feature files across both Cypress and Selenium. If anyone has ideas on this, please comment below with your suggestion 😃

It depends on how far you take this strategy and to decide if having duplicate test coverage is worth it to you. For us, having at least happy-path end-to-end test coverage in I.E.11 gave us a huge amount of confidence when deploying so the cost was worth it. In my opinion, it isn't as bad as it seems, our Cypress tests cover Chromium-based browsers (with Firefox support coming soon) and our Selenium tests cover I.E.11. With I.E.11 being phased out more and more, even in the enterprise, the need for Selenium will go away and the need for Cypress will get even larger.

Bonus: Typescript Support and Code Coverage

All of the libraries and modules I have mentioned previously come with Typescript support. Getting Typescript to work with Cypress doesn't require many configs and is totally worth it in the long run. All you will need are Webpack, TS config, plugin files which integrate with Cypress. A good guide provided by Cypress is here.

I know a lot of people wonder about code coverage and generating reports, Cypress can do that as well! Again, there is a nice plugin that lets you do it. The one caveat is that it will attach coverage counters to your code so running your tests will be slower and may not mimic production. A good strategy here is to just generate them locally once in a while to see how you are doing.

If your backend and frontend are in Typescript, a cool idea is having code coverage running in both apps when Cypress runs. You can then see the coverage across your entire app!

This shell is meant for most people, Fish stands for Friendly Interactive Shell, that's why I can recommend it to anyone I work with. They have a very detailed design document. It's not meant for the likes of system admins who are constantly logging into multiple servers a day. It will never be the default installed shell on most operating systems.

Once you install it, brew install fish, you are off to the races. You have a shell where you can become super productive and your favourite tools work as intended. It doesn't try to be the best at everything, but nails the essential core features which make the user experience extremely enjoyable.

TLDR

• Syntax highlighting
• Inline auto-suggestions based on history
• Tab completion using man page data
• Intuitive wildcard support
• Web based configuration

Let's break it down

Syntax highlighting

My worst memories of bash come from the absence of this feature, syntax highlighting. A simple thing which makes you think, "wow, now I am using a shell from the 90's"! You can notice it working in the below gif when I try to go to folder_that_doesnt_exist, the text turns red. The text then turns blue when it's a valid command.

Inline auto-suggestions based on history

Smart auto-suggestions are seldom seen, let alone built-in. Instead of just beating the competition, the Fish team thought to demolish it. Using the history of your commands, it suggests commands which you can complete with the right-arrow key. You can also, as I do this gif, auto-complete one word or folder at a time with option + right-arrow key.

Fun fact, if search results are huge, Fish shell will paginate!

Tab completion using man page data

This is because Fish knows how to parse CLI tool man pages in many different formats. Git, Docker CLI, package.json, you name it, most commands you try, it will have auto-completions for it.

You can use tab to get all the options.

All npm scripts for this package, with values of what they actually run, IN THE TERMINAL WUT

Intuitive wildcard support

In bash, I never liked having to use different flags for selecting files or contents of a folder.

Regularly, this would be done with:

rm -r folder_1

I have always been a fan of familiarity, and wildcards are just that. You can use them in any command filter down the exact files you need with ease.

e.g.

ls *.jpg

How I feel while using Fish

Web based configuration

Type in:

web_config

and you get an entire website dedicated to messing around with any config you do need to touch.

A tiny customization needed to go the extra mile

There aren't a lot of extra packages needed for Fish. Personally, I only use 2, which is wild because at one point I know my Oh-My-Zsh plugins were past 10.

Oh My Fish

A homage to the great Oh My Zsh, omf is the most popular package manager for Fish. I use this to install just two packages, one for nvm and one for spacefish.

SpaceFish

Special mention to Spacefish for being the best shell prompt I have ever used. Support for showing:

• Current Git branch and rich repo status
• Current Node.js version, through nvm
• Package version, if there is a package in the current directory (package.json for example)

Spacefish example

Config file

You also have access to a config file at .config/fish/config.sh. This is where you can set aliases up or set some extra path extensions.

Caveats

Not being POSIX compliant can scare some developers away. But really in my three years of usage (mostly Node.js, javascript, ruby, e.t.c.), I have not encountered any issues. Some commands I get from the internet which are Bash specific, I'll just exit and then come back to Fish when I finish. This Stackoverflow post goes into it more if you are so inclined.

But it's easy to be compatible...

Say you have a bash script to run, with Fish you still can:

bash script.sh

Another tip is that you can put this at the top of the file:

#!/usr/bin/env bash

and then make sure its an executable:

chmod +x script.sh

and voila, you can run it as a regular script:

./script.sh

Resources:

• Fish Shell Website
• Fish Shell Syntax Highlighting
• Fish Shell Autosuggestions
• Try out the Fish Shell tutorial online
• Oh My Fish Package Manager
• NVM wrapper plugin
• Spacefish Fish Shell Theme
• List of awesome Fish related software
• Fisher, another package manager with a file-based extension config
  • A friends fishfile for fisher
• Support Bash scripts in Fish

Like this post? Consider buying me a coffee to support me writing more.

Want to receive quarterly emails with new posts? Signup for my newsletter

{% instagram BvmFY7DgPyg %}

With our honeymoon in Kenya on the horizon, we set out to book a room. Consulting my aunt who had been to Kenya years ago, she stayed here and had no difficulties booking. It came to our surprise when we heard that this place was fully booked a year or two in advance.

The sudden popularity had to stem from something. A little researched showed this place being recently Ellen’ed.

{% instagram BjQFBIDhsH_ %}

Damn it, Ellen.

Initially, we checked their website to see if the dates we would be in Kenya were available, no luck. We then emailed the manor and again, no beauno, we were told we were put on their “waitlist”. Likely competing with other people on the waitlist, and our trip only being a few months away, me and my wife's hopes drew thin.

The search for solutions

The website they were using to show availability was read-only, no functionality to book rooms.

Calling and email were the only way to reach them, a slow and arduous process. I assumed that when a date became free, their website would update first and then they would start contacting waitlist members. This way, they would still get bookings if people fell through.

Assumptions

What I assumed next is that if we were to contact them the day the room became available, likely we would bypass the waitlist. But checking the website every hour was not going to fun.

I put my programmer pants on and thought that this would be a good use case for a good-ol web-scrapper, jazz hands. Hit the site every 30 min and SMS both my phone and my wife’s so that we could give them a call. Unlikely that this 1990's Kenyan website had protection against bots.

What looked like a simple table turned out to be a simple table:

// Example of a unbooked day HTML node

<td
  width="25"
  unselectable="on"
  ab="0"
  style="border-top: none; "
  name="WB15:Salas Camp:Keekorok Honeymoon
  Tent-Tent 1:0*:1:11e8485f8b9898cc8de0ac1f6b165406:0"
  id="WB15:07:28:2019"
  darkness="0"
  onmousedown="mouseDownFunction(arguments[0]);"
  onmouseup="cMouseUp(arguments[0]);"
  onmouseover="mouseOverFunction(arguments[0]);"
  class="overbooking calIndicator0"
>
  1
</td>

This is what I needed to find, if it the node text was 1, it was available.

After investigating the simple html structure, I started writing the Node.js service to scrap it. I stumbled upon an NPM module, crawler, which gave me all I needed out of the box.

const Crawler = require("crawler");

const startCrawler = async () => {
  return new Promise(resolve => {
    const c = new Crawler({
      maxConnections: 10,
      callback: (error, res, done) => {
        if (error) {
          console.log(error);
          throw new Error(
            `Error with sending request to website! ${JSON.stringify(error)}`
          );
        }
        const $ = res.$;
        // get the table of bookings
        const results = $("#tblCalendar tbody tr").slice(12, 17);
        done();
        // return the results
        resolve(results);
      }
    });
    // hit giraffe manors website
    c.queue(
      "http://thesafaricollection.resrequest.com/reservation.php?20+2019-02-08" +
        "+RS12:RS14:RS16:WB656:RS2274+15:20:30:25++WB5++n/a++true+true+0+0"
    );
  });
};

This took a bit of debugging but now I had the HTML from Giraffe Manors website to play around with.

Next up, I went searching through the results with an NPM package called cheerio.

const parseResults = async () => {
  let availability = false;

  // get HMTL
  const results = await startCrawler();

  for (let x = 0; x < results.length; x++) {
    // Feb 13th - Feb 20th
    const validDates = cheerio(results[x]).find("td").slice(7, 14);
    // See if any of the dates are not booked
    for (let y = 0; y < validDates.length; y++) {
      if (parseInt(validDates[y].children[0].data, 10) === 1) {
        availability = true;
      }
    }
  }
  ...

Now comes the interesting part, SMS text my wife when the room show as available. I used Twilio for this but many other services exist. This required setting up a free account, I know I wouldn't be sending more than a few SMS messages.

  ...
  // send text message if availability
  if (availability) {
    // Your Account Sid and Auth Token from twilio.com/console
    const accountSid = process.env.ACCOUND_SID;
    const authToken = process.env.AUTH_TOKEN;
    const twilio = require("twilio");
    const client = twilio(accountSid, authToken);

    client.messages
      .create({
        body: "Giraffe manor is available for our dates!",
        from: process.env.SMS_FROM,
        to: process.env.SMS_TO
      })
      .then(message => console.log(`Sent a text! ${message.sid}`))
      .done();
    return;
  }
  console.log("No availability!");
}

After testing with a few dates that were unbooked, it worked! Now to schedule it to run every 5 minutes (because why not?).

const schedule = require("node-schedule");

schedule.scheduleJob("*/5 * * * *", () => {
  console.log("Running availability checker!");
  try {
    main();
  } catch (e) {
    console.log(`Error! ${JSON.stringify(e)}`);
  }
});

To host and run the code, I chose Heroku as I have experience with it and knew the free tier would work for what I needed. I have no idea how their free tier supports background service jobs but anyways.

A couple of weeks later, (I actually forgot it was running), my wife received the text to her phone! We called them immediately and got it! Seemingly bypassing the waitlist, just like we had hoped. She got a barrage of texts and used up my free tier on Twilio as I didn't write a stop method when it found an available room 🤣

I particularly liked doing this because it's not often I code to solve a problem in my life but I thought it would be worth it for pictures like this:

This was one example of how I used my programming skills for a "real" world problem. I would love to hear a problem you may have solved, comment here.

The code

Like this post? Consider buying me a coffee to support me writing more.

Want to receive quarterly emails with new posts? Signup for my newsletter

For use with our frontend applications, we opted for Apollo Client with React which seems to be the one true GraphQL client at this point. As the library is fairly new (the javascript ecosystem moves fast, who knew?) we have experienced our fair share of pains and troubleshooting. Some of which included:

• Using Apollo Local State on a large codebase and running in production
• Integration with Server Side Rendering (Next.js)
• Generating Typescript types on a stitched schema

This article focuses on the first point, managing frontend local state. When looking for state management solutions, Apollo Local State (formally apollo-link-state) popped up. A couple of reasons led us to using this library:

• The data models and store structure can be shared between the data fetching cache and the local state management cache
  • This leads to sharing of Typescript types as well 😉
• Actions are performed with mutations, something that previous GraphQL users already understand.
• Staying within the Apollo ecosystem meant smooth integration with existing tools, meaning less overhead for developers.
• Backed by Apollo meant that support would be there for a significant amount of time.

Another good sign was this very attractive article which explains the advantages of keeping your local state close to the GraphQL schema vs using something like Redux.

Here comes the pain

A pattern established by Flux (the paradigm, not the library), has you splitting up actions for every event which happens in your app. User clicked a button, action is triggered, user scrolls down a certain length, action is triggered. Your app can observe these actions and manipulate the state accordingly. With Apollo Local State Mutations, this becomes much more intentional. No observability is given, each action is directly related to a resolver.

For example, you want to update a name on an issue (think GitHub just for examples sake) in the cache, this is Apollo’s term for state:

const IssueContainer: React.FC<{ issue: GithubIssue }> = ({ issue }) => (
  <UpdateIssueNameMutation mutation={UPDATE_ISSUE_NAME}>
    {(updateIssuename) => (
      <IssueFields
        issue={issue}
        onChange={(name) => {
          updateIssuename({
            variables: {
              input: {
                id: issue.id,
                name,
              },
            },
          });
        }}
      />
    )}
  </UpdateIssueNameMutation>
);

For this above mutation, here is an example of what we would need to write in Typescript, I break down what’s going on in the comments:

import gql from "graphql-tag";
import { IFieldResolver } from "graphql-tools";

import { ISSUE_PARTS } from "../issues";
import { IssueParts, UpdateIssueNameVariables } from "../../graphql-types";
import { ResolverContext } from ".";

export const UPDATE_ISSUE_NAME = gql`
  mutation UpdateIssueName($input: UpdateIssueNameInput!) {
    updateIssueName(input: $input) @client
  }
`;

const ISSUE_FRAGMENT = gql`
  ${ISSUE_PARTS}

  fragment IssueParts on Issue {
    id @client
    name @client
  }
`;

/**
 * Updates the name of a Github issue.
 **/
const updateIssuename: IFieldResolver<void, ResolverContext, any> = (
  _obj,
  args: UpdateIssueNameVariables,
  context
) => {
  const { input } = args;
  const { cache, getCacheKey } = context;

  // 1. Get the id of the object in the cache using the actual issue id
  const id = getCacheKey({
    __typename: "Issue",
    id: input.id,
  });

  // 2. Get the data from the cache
  const issue: IssueParts | null = cache.readFragment({
    fragment: ISSUE_FRAGMENT,
    fragmentName: "IssueParts",
    id,
  });
  if (!issue) {
    return null;
  }

  // 3. Update the data locally
  const updatedIssue = {
    ...issue,
    name: input.name,
  };

  // 4. Write the data back to the cache
  cache.writeFragment({
    fragment: ISSUE_FRAGMENT,
    fragmentName: "IssueParts",
    id,
    data: updatedIssue,
  });
  return null;
};

export default updateIssuename;

That’s 60 lines to update a single attribute on one data entity in the cache. After doing a couple of these, you will start to pull your hair out. Mutations as is, don’t do a whole lot for you, this results in a lot of boilerplate. Having all of the boilerplate code is not ideal, it leads to more bugs and thus more tests need to be written to avoid those bugs.

Looking for patterns

After writing about ten of these with plans to write a lot more, we wrote a small Yeoman generator to speed up the process. This made writing them a lot faster but didn’t solve the bloat in our codebase. Every mutation ended up doing the same thing as described in the comments above:

1. Get the id of the object in the cache using the actual entity id
2. Get the data from the cache
3. Update the data locally
4. Write the data back to the cache

The solution

Naturally, we wrote a helper which would help us refactor our resolvers.

import { IFieldResolver } from "graphql-tools";
import { DocumentNode } from "graphql";

import { ResolverContext } from ".";

interface InputVariablesShape<TInput> {
  input: TInput;
}

/**
 * Creates a client-side mutation resolver which reads one item from the cache,
 * mutates it using the given mutation, then writes it back to the cache.
 * @param reducer Func which mutates data and returns it. Must be a pure function.
 * @param fragment Used to read/write data to apollo-link-state.
 * @param fragmentName Name of fragment inside
 * @param getId A func which returns the id of the entity to be used in the `reducer` func.
 */
export const createResolver = <InputShape, EntityType>(
  reducer: (entity: EntityType, input: InputShape) => EntityType,
  fragment: DocumentNode,
  fragmentName: string,
  getId: (input: InputShape) => string
) => {
  const resolver: IFieldResolver<void, ResolverContext, any> = (
    _obj,
    args: InputVariablesShape<InputShape>,
    { cache, getCacheKey }
  ) => {
    const { input } = args;

    // 1. Get the id of the object in the cache using `getId`
    const id = getCacheKey({ id: getId(input) });

    // 2. Get the data from the cache
    const entity: EntityType | null = cache.readFragment({
      fragment,
      fragmentName,
      id,
    });

    if (!entity) {
      return null;
    }

    // 3. Update the data locally
    const newEntity: EntityType = reducer(entity, input);

    // 4. Write the data back to the cache
    cache.writeFragment({
      fragment,
      fragmentName,
      id,
      data: newEntity,
    });
    return null;
  };

  return resolver;
};

This resulted in the resolver code becoming a small pure func:

const fragment = gql`
  ${ISSUE_PARTS}

  fragment BasicIssueParts on BasicIssue {
    ... on Node {
      id
    }
    parameters {
      id
      value
    }
  }
`;

export const reducer = (
  issue: IssueParts,
  input: UpdateIssueNameInput
): IssueParts => ({
  ...issue,
  name: input.name,
});

export default createResolver(
  reducer,
  fragment,
  "IssueParts",
  (input: UpdateIssueNameInput) => {
    return input.id;
  }
);

A two-line resolver which does the same as before. This covered about 90% of our use-cases!

The road ahead

If you found this blog post helpful, don’t hesitate to steal the gist of this code.

Goals

Let's say we want do some specialized work, scanning an image to extract text for instance. This is a situation where a job queue could come in handy, this work is being done in the background, away from a user facing interface.

• Get image from user
• Queue job with image attached
• Job gets worked on
• Job results are sent back to app database

Two popular packages in the wild which could help you do the forementioned work are DelayedJob and Celery. These allow you to manage jobs with a fast key-store like Redis. These assume the processing of the job and the job queue live in the same service. If you have one service performing a task, e.g. the image processor, and another service which acts as a job queue, we cannot use these traditional constructs.

This (Diagram 1)

versus

This (Diagram 2)

A Solution

Me and my coworkers found ourselves in this situation and when searching for answers, we found Bull might suffice. Keeping it 2018, this Node.js package is lightning fast, built to work with Redis and has an active community. It didn't quite fit our needs at first as it processed jobs in the same app as the queue-mechanism, see diagram 1. This is fine for traditional apps, but for our setup we needed to manage jobs across systems (see diagram 2). We needed to make this work in an async fashion where the worker may not be in the same repo or service as the service running Bull itself.

We need to think about how we want to manage a jobs life-cycle. Good thing someone contributed a diagram quite recently to the projects Github.

Bull's Job Lifecycle Diagram

Bull had a simple way to define the processing logic (refer to diagram 1), what a job does when in the active queue:

queue.process(async () => {
  doWork()
})

This way, whenever a job came into a waiting queue, Bull knew how to process it and throw it to the completed queue. Right now, Bull managed all the state transitions on it's own, we need to switch to manual. You may be thinking, "to work in this new fashion, how about we just don't define this process method?", we tried this, and it worked!. Forward into the weeds we go.

but for our setup we needed to manage jobs across systems

After digging into the code more, Bull defines state transition methods on two simple objects, Job and Queue.

After researching, the methods to do manual state transitions were private. It means that the authors didn't write these methods to be used publicly. This makes sense as Bull was never designed to do what we want to do with it. What do we need to do to make these public? After some more digging, we found someone else trying to do the same thing as us.

The issue can be found here.

Just using the private functions as is would have been fine but we are professional developers.

I would recommend that you write a few unit tests specifically for testing the code using the private functions... - @manast

The maintainer had a great suggestion, write unit tests for the private functions. The next best thing for this would be to at least write documentation for the functions so that they are understood by the community and strengthened their viability to be used publicly. And that's what we did.

Open Source Bonus

For the actual pattern we described at the beginning (diagram 2), an addition to the reference docs were added to make this a viable pattern. Making this a known pattern encourages usage of the feature and possibly leads to other users finding issues when using in production. Typescript types were also available so we updated those as well. After using it for some time (processing approx. 500k jobs), we found a bug and were able to easily fix it using our extended knowledge of the package. Talk about bringing a third class feature to first class!

I am very happy with the outcome of the project as not only did we satisfy our requirements but also made open source contributions. This led to us understanding the packages internals and also led to us being able to easily add features for our use case. Having an active maintainer on the project who knew the ins and outside also made the entire process run smoothly.

E.g. of pattern matching. This and many other features of the language make it expressive and easy to read.

Quick Facts for the T.L.D.R. in you

• Elixir is simply syntax on top of Erlang, the battle-tested language built on top of the BEAM VM

• The syntax is similar to Ruby so learning the syntax is easy and quick, especially for developers familiar with it

• Did I mention it's FUNCTIONAL! (Pure, functional programming IMO is worth the investment cognitively, hit this link for how Elixir utilizes it)

One of the benefits of learning a recently-created programming language is that it's built on top of existing best practices. This happens when the creators spend time thinking about what problems other developers face regularly. "State management is hard", "it's hard to have zero time deployments of new code", "it's hard to maintain my systems", something every developer thinks. Elixir wants to make these problems less hairy and does so using functional methodologies wrapped around a VM which puts distributed/concurrent programming as a first-class citizen. Elixir for example was built by developers who saw the productivity of the Ruby syntax, the maintainability of functional programming and the scalability of Erlang. These features of the language make it a compelling showcase of what a language recently built can be, as showcased in the pattern matching example above.

Elixir for example was built by developers who saw the productivity of the Ruby syntax, the maintainability of functional programming and the scalability of Erlang.

Wires connecting to wires

OTP in the anime-flesh

The rock solid foundation of Elixir is built on top of a library named OTP. OTP is an elegant way to handle all of the problems that arise in distributed programming, think work across nodes, handling async messages, etc. It not only is a library of functions but also a paradigm to work within. This keeps things consistent across systems and large teams. Instead of a single process handling your entire app (think Node.js), many isolated processes make up an Elixir app. These processes communicate to each other using messages. This unlocks a lot of cool features, processes can now live across machines as messages can only be immutable, no pointers allowed.

The critic inside you will say the potential downfalls of using such a new language is that it isn't battle-tested. Usually this is a valid criticism, such is not the case for Elixir. The VM Elixir it's built on top of is hella old. The initial open-source release of Erlang was in 1998, and Ericsson was using it in-house for a long time before that. Used by telecom networks, these were critical services which could not afford to have downtime. For example, that's how the very cool hot-code-release feature came to be which enabled developers to release new Erlang/Elixir code without taking down servers.

My Experience

A candid photo of me reading Elixir in Action

Last year, a coworker invited me to join his book club. "Lets learn this new language." I had heard it was the new hotness so I said, "sure!". We would take a couple of hours every month to go over a chapter in the book, Elixir in Action. Initially, it was intimidating to join as I was vastly junior compared to the other members of the group but I gave it a shot. What followed was lots of great discussions and insight into topics I haven't dove into before. I am appreciating my former self for agreeing to join as not only did I learn a lot, I connected with coworkers in the company I would have never connected with otherwise. It helped me through Flipp's adoption of Event Driven Systems (think Kafka) by exposing me to good practices when managing state between processes. Keeping processes small, pure and functional is good-sound engineering practice and are the pillars of how Elixir works. I didn't need anything to build immediately or an assignment to finish, I learned for the joy of learning and got a lot out of it.

Common comments and questions

My team is not going to be happy that after learning 3 Javascript frameworks in the past week, they have to learn this.

Once you start building things that have to scale or need to handle millions of requests, your on-call tickets increase. The reason for this is usually you can't predict traffic at that scale, push notifications go out for a new feature and everyone starts hitting your API. How do you handle this currently, with something like Node or Ruby? You just increase your box numbers and then decrease them after the load is done. This gets expensive and developers should not just be throwing money at something to solve a problem. Erlang VM processes (different than the traditional process) are a fixed size, this is mega. To a degree, this essentially solves this problem. Knowing how much memory processes are, gives you god-like abilities. The VM can tell the server precisely how much memory it may potentially use. Instead of falling over and the box restarting, you could respond to the client with HTTP Status Code 429 for example. No more unexpected memory loads at 1AM waking up developers!

Okay, this is dope, how are errors handled?

Errors are a first class citizen in Elixir. Processes are small and isolated so when an error is thrown, the entire app process doesn't have to dump it's stack, just the isolated process. When errors do happen, they are easier to debug as the process code is small (by Elixir convention). Processes are so small that every process gets a monitor (another OTP blessing), which can run some code when a process dies. An example monitor could restart the process for example so that it could accept more messages.

Everyone gets a monitor

Also, it's very neat that there is a proposal for pattern matching in Javascipt. Obvious proof that everyone is drinking the ... wait for it ... Elixir.

🚒

The road forward

I hope this introduction shows you some of the powers of Elixir and encourages you to learn more. I just scratched the service of what is possible with the BEAM VM. I leave you with this graph showing Elixir's popularity on Stackoverflow compared to other popular languages:

Perspective

The trend is upwards but it still has a long way to go for becoming somewhat mainstream.

Moving forward, I plan is just to write more and more Elixir code and get more comfortable with it. HackerRank has Elixir as an environment so it has been a great resource to practice the syntax. One of the next things I want to do is start creating something in Phoenix.

Another resource I used in my learning journey was the Elixir Toronto Meetup Group on Meetup.

Reading resources

The book we read during the book club was called Elixir In Action. A very good book which goes through the entire language and its features, in detail. The beginning is quite slow but as you start to wrap your brain around syntax, it soon becomes super interesting.

Elixir in Action

This is another book I started which is much more approachable. It's a fun book which goes over the main features of why Elixir is a compelling language. This is a heart-pumper as it really just skims the surface.

The Little Elixir & OTP Guidebook

Originally posted on my blog.

Like this post? Consider buying me a coffee to support me writing more.

Want to receive quarterly emails with new posts? Signup for my newsletter