Cloudflare Workers let you host pages and run code without managing servers. Unlike traditional servers placed in a single or a few locations, the deployed static assets and code are mirrored around the globe in the data centers shown as blue dots below. Naturally, this offers better latencies, scalability and robustness.

Their developer platform also extends beyond “Workers” (the compute part) and include storage, databases, queues, AI and lots of other developer tooling. The whole with a generous free tier and reasonable pricing beyond that.

Why am I writing this? I find it fairly good, had a good experience with it, and that’s why I will present it here. This article is not sponsored in any way. I just think it’s somehow a responsibility of developers to communicate about the tools they use in order to keep their ecosystem lively. I’ve seen too much good stuff getting abandoned because there was no “buzz”.

The benefits of using Cloudflare Workers is:

• Great latencies worldwide

• Unlimited scalability

• No servers to take care of

• Further tooling for data, files, AI, etc.

• GitHub pull requests preview URLs

• Free tier good enough for most hobby projects

When not to use it

Like every tool, it has use cases for which it shines and others it is not suited for. This is important to grasp and understanding the underlying technology helps tremendously. Basically, in loads your whole app bundled as a script and evaluates it on the fly. It’s fast and works wonderfully if your API and used frameworks are slim and minimalistic. However, it would be ill-advised in following use cases:

• Large complex apps
  The cost of evaluating your API / SSR script will grow as your app grows. The larger it becomes, the more inefficient its invocation as a whole will become. There are also some limits how large your “script” can be. Although it has been raised multiple times in the past, the fact that this is extremely inefficient will always remain. Thus, be careful when picking dependencies/frameworks since they can quickly bloat your codebase.

• Heavy resource consumption
  Due to its nature, it is not suited to compute stuff requiring large amounts of CPU/RAM/time like statistic models or scientific computation. Large caches are problematic too. Waiting for long-running async server-side requests is OK though, the execution is suspended in-between and do not count towards execution time.

• Long-lived connections

  That’s also problematic. You should rather use polling than keeping connections open.

In other words: “The slimmer, the better!”

It’s kind of difficult to say what’s small enough and when it becomes too large. This is rather suited for small self-contained microservices of modest size. Even debugging using breakpoint might turn out challenging. For such larger applications, traditional server deployments would be more suited.

What will we build?

A “Quote of the Day” Web application.

The purpose is not to build something big, but rather a simple proof-of-concept. The quotes will be stored in a KV store and fetched Client-side. That way, we can measure how fast the whole works and if it lives up to the expectations.

The default version of https://quoted.day is available in two flavours:

• https://quoted.day/spa: a static page, fetching the quote text/author asynchronously

• https://quoted.day/ssr: Server-Side-Rendering, rendering the page with the quote on the server

I swapped which one is the default from time to time to perform experiments. Performance (latency) may vary depending where you are located and whether what you fetch is “hot” or “cold”. Before we delve into details on how to build such an app, let’s take a look at the performance we can expect.

Benchmarking latencies worldwide

Unlike the internal Cloudflare latency measures, measured “inside” the worker and therefore quite optimistic, we will look at the “real” external latency thanks to the great tool https://www.openstatus.dev/play/checker .

Thanks to that, we can obtain a pretty good idea of the overall latencies that can be observed all over the world. Note however that Australia, Asia and Africa may have rather erratic latencies that “jump” sometimes.

We will also benchmark multiple things separately:

• Static assets

• Stateless functions

• Hot KV read

• Cold KV read

• KV writes

Also, every case will get “two passes”, to hopefully fill caches on the way, and only record the second one.

Static assets

This was obtained by fetch the main page at https://quoted.day/spa

Stateless function

his is obtained by fetching the endpoint https://quoted.day/api/time which simply returns the current time.

Hot KV read

This is obtained by fetching a fixed quote from the KV store using the endpoint https://quoted.day/api/quote/123

Cold KV read

This is obtained by fetching a random quote from the KV store using the endpoint https://quoted.day/api/quote

Note that each call will cache the result for a day at the edge location, resulting in possibly turning cold reads into hot reads as traffic increases.

KV writes

This is obtained by fetching quoted.day/api/bump-counter which creates a temporary KV pair with an expiration time of 10 minutes. It kind of emulates the concept of initiating a “session”.

SSR Page with KV cold reads

Lastly, in this test, we combine the reading a random quote (that usually results in a cold KV read) and renders it server-side in a page.

Observations

In is interesting to see how you can infer how the KV works just by watching the numbers. It appears the KV store is not actively replicated, but rather KV pairs are copied “on-demand” at remote locations. When cached (by default 1 minute), subsequent reads are fast. The latencies of such “hot” KV pairs are pretty good overall. No complains here. How long the pair remains cached there can also be configured using the cacheTtl parameter during the KV get request. However, the downside of increasing that value is that this cached copy do not reflect changes / updates triggered from other locations during that time.

Unsurprisingly, cold reads have worse latencies. The other thing you can infer from the numbers is that there seem to be an “origin location”, and cold reads latencies increase proportionally according to the distance to this location. Therefore, pay attention “where” you create the KV store, as it impacts all future latencies around the globe. Note that workers KV might change in the future, this is merely an observation of its state right now.

While read operations are OK, the write operations are rather disappointing right now. I expected it to have great latencies too, writing to the “edge” and letting the propagation take place asynchronously, but it is the opposite. Writes appear to communicates with the “origin” storage. The time it takes to set a value gets higher the further away you are from where you created the KV store. This is kind of bad news, because setting/updating values is a pretty common operation, for example to authenticate users. Dear Cloudflare team, I hope you improve that part in the future.

A word of caution

If you develop your webapp, publish it and take a look at it, you will probably not even notice the bad latencies. You will face the optimal latencies with the origin KV store being near you. However, someone at the other end of the planet will have an uglier experience. If that person has a handful of cache misses or writes, the response time might quickly climb into a few seconds before the response arrives. That is not how I would expect a “distributed” KV store to behave. Let us be clear, right now this behaves more like a centralized KV store with on-demand cached copies at the edge.

Quite ironically, it basically feels more like a traditional single-location database right now (+caches). While latencies of a single cache miss or a single write is not dramatic, it can quickly pile up with multiple calls and especially write-heavy webapps risk facing increased “sluggishness” depending on their location. Here as well, being “minimalistic” regarding KV calls should be taken to heart during the conception of the webapp using workers.

Lastly, there was one more setting available in the Worker: “Default Placement” vs “Smart Placement”. I tried both but I did not see noticeable changes within the latencies. I think it’s due to the fact that there is a single KV store call and that it takes time and traffic to gather telemetry and adjust the placement of workers. It might be great, but for this experiment, it had no effect at all.

Single-Page-Applications vs Server-Side-Rendering

Here as well, one is not universally better or worse than the other and the answer which one to use is “it depends”.

Besides strong differences regarding frameworks and overall architecture, it also has practical fundamental differences for the end user. It’s also fascinating to see history repeating itself, where the internet first started with server rendered pages, than single-page-application with data fetching took over and a resurgence of SSR, just like in the past, just with new tech stacks.

SSR is actually the easiest one to explain: you fetch all the required data server side, put everything in a template and return the resulting page to the end user. It takes a bit of time and processing power server-side, is not cachable, but the client gets a “finished” page.

The SPA does the opposite. Although the HTML/CSS/JS is static and cached (hence quickly fetched), the resources are typically much larger due to all the client-side javascript libs needed. Then starts the heavy lifting, where data is fetched and the page rendered, typically while showing a loading spinner. As a result, the total time to render the page is longer.

However, interacting with the SPA is typically smoother afterwards, because interactions just exchange data with the server and make local changes to the page. In contrast, SSR means navigating and loading a new page. Hence, the choice whether SPA or SSR is more suited depends on how “interactive” the page/app should be.

As a rule of thumb, if it’s more like a static “web page”, go for SSR, if it’s more like an interactive “web app”, go for SPA.

Lastly, the nice thing about Astro, picked here as illustrative example, is that the whole spectrum is possible: static pages, SPA and SSR.

Sources

The source code of this experiment is here: https://github.com/dagnelies/quoted-day

If you have a Github and a Cloudflare Account, you can also fork & deploy by clicking here:

If the button doesn’t work, here it is as link instead: https://deploy.workers.cloudflare.com/?url=https://github.com/dagnelies/quoted-day

It will fork the GitHub repository and deploy it on an internal URL so that you can preview it. Afterwards, you can edit the code and it will auto-deploy it, etc.

Note that the example references a KV store that is mine. So you will have to create your own KV store named and swap the QUOTES KV id in the wrangler.json file with yours. You will also have to initially fill it with quotes if you want to reproduce the example. Luckily, there are scripts in the package.json to do just that.

Everything beyond this point would deserve a tutorial on its own. This was merely the result of an experiment, how the latencies hold up and some insights on the platform. Enjoy!

At first sight, Gradle looks cool:

• Their website looks way nicer and polished than Maven's one

• The syntax is much more compact than Maven's verbose XML

• Gradle is "newer" while Maven is "older"

• Gradle is much faster (according to them)

No wonder people pick it up when faced with uncertainty and just wanna get started. So now, let me tell what's wrong with Gradle IMHO and why Maven is still the better option, even so many years later.

Configuration vs Scripting

Basically, the pom.xml that you define in Maven is a "configuration". You define the name, the version, the list of dependencies, etc. Since it follows a specific schema, with a set of properties to define, you can also look at it visually through a UI for example. It's a declarative definition of your library/app.

On the opposite, a Gradle build script is exactly that: a script. It's using the Groovy language, or recently also Kotlin, to let you write anything you want. Let it sink in, you use a programming language to define what the build should do. You can import other scripts, send a HTTP request to check the current weather and insert a funny UI generated picture in your build artifact.

While they have many aspects in common, they nature ("configuration" vs "script") is what differentiates them fundamentally.

You may think: "Isn't it great if I can do anything with that build script?! It's ultimate freedom!". That is right, but this boon is also a curse.

When I see a Maven project, with a pom.xml, I know what it does and where to find what. It's always the same. Directories, commands to run, changing the version, whatever, it's the same for all maven projects.

When I see a Gradle project, I have no idea what the build script does. If you don't have a clear documentation ready, you'll have to dive into the build script to actually discover and try to understand what it exactly does.

The price of freedom

It's not rare that you need something specific in your build. In both Maven and Gradle, it's possible to do so, but here also their approach is opposite.

In Gradle, it's straightforward. Since it's scripting, just write whatever you want, you can do anything very easily. Your own build stages, calling functions, using variables, importing some other scripts, whatever. It's easy.

In Maven, it's the opposite. Adding something custom is more difficult. You will have to use a plugin to enable the specific functionality, or even write a plugin yourself if really necessary. While writing a plugin is definitely more work, this kind of also enforces reusability though.

The takeaway here is the same as before. While Maven builds tend to always follow the same build stages and conventions, Gradle builds tend to become more and more complex and customized over time, because it's so easy to "just add a few lines" to the build script. Look at it after a few years and the Maven pom.xml is likely almost as readable as the first days while the Gradle build.gradle script became rocket science.

As an exercise, I picked a random gradle.build file from another team at work to look at it. It had over thousand lines and the few dependencies it had were externalized in another file and combined in a fancy way.

On the opposite, pick any Maven project, and the list of dependencies will always be in the same place, in the <dependencies>...</dependencies> tag.

History repeating itself. As a side note, it's interesting to see that in the very early days of Java, before maven was born, build scripts were the norm. In the beginning they were plain shell scripts invoking javac ... to compile the source code, packaging it, etc. Then came "ant" to do the same, in a bit more structured way but still tended to become customized and complex over time. Then one day came the idea to use a more declarative approach, by defining a project and its dependencies while letting the tool take care of how it is build. Maven was born. Then, some day, Gradle was born, because "I want to customize stuff".

Gradle lies in your face

Now, this is a little grudge of mine against Gradle's marketing habits. When going on their website, they will feature a "Gradle vs Maven" comparison claiming that Gradle is "oh so much faster" than Maven and the following picture.

Now, let's take a closer look...

First, what's shocking is that a "Clean build with tests" is so much faster than the original build! It's almost instant! Including tests! ...let me get this straight: this is not "clean" at all. It's just doing nothing. I find Maven much more sensible in that case, it actually rebuilds everything from scratch. To go a little bit further, a "build" in Maven will just check for changes and compile changed files, which would result in a similar figure, while a "clean build" will remove the whole directory and re-build everything. I find this should be the expected behavior unlike Gradle's "clean build" not cleaning anything. After all, the aim of a clean build is usually to fix issues due to some undesirable thing lying around in the build directory, for whatever reason.

Then, let's look at the normal case: is Gradle really twice as fast? Well, here is another question for you: who compiles the source code? ...got an idea? Well, it's the javac compiler from the JDK, it's not the build tool! So why would Gradle be twice as fast?! Here is the trick: Gradle runs the tests in parallel while Maven do it sequentially. That is the reason! Gradle ain't faster or anything, it just runs the tests in parallel. I dislike this default. It's just a question of time until you get tests having side-effects and race conditions. Then you'll obtain "Heisenberg tests" succeeding sometimes and failing sometimes, depending on how their executions overlap. You'll wonder why and waste lots of time investigating the issue. Moreover, it usually runs in a background jobs after commits anyway.

Now, while I dislike Gradle's defaults, what I'm really annoyed about is how they distort the truth. They should say "we run tests in parallel by default and our 'clean' does nothing instead!". That should have been the correct way to put it instead of using their misleading statements insinuating that they compile faster.

Gradle is not simple

For Maven, the scope of dependencies is relatively straightforward:

• Compile (the "usual" dependency)

• Test (for tests)

• Provided (provided at runtime by JDK or a container)

• Runtime (quite rare. For drivers or alike available at runtime but not for compiling)

It's enough and I never needed anything else.

Gradle on the other hand has lots of scopes:

• api

• implementation

• compileOnly

• compileOnlyApi

• runtimeOnly

• testImplementation

• testCompileOnly

• testRuntimeOnly

• ...a few more deprecated scopes

• ...a few more classpath scopes

• ...you can also extend and combine scopes

Well, you basically get it. Gradle is "super-customizable", so much that you often wonder what it exactly does or that you make a mistake without realizing. Gradle sells it as "Maven has few, built-in dependency scopes, which forces awkward module architectures" but IMHO it's Gradle which is confusing and overcomplexified here while Maven has exactly what's actually required.

That is just the tip of the iceberg. But basically Gradle is super-customizable while maven favors conventions. No wonder Gradle is also a company that thrives with support and training. If it was simple, such things would not sell.

Gradle needs maven, but not the other way round

Every single library in the Maven Central Repository must have pom.xml. It's the declarative definition of the library containing name, version, license, etc, and most importantly the list of its dependencies. Without a Maven pom.xml there would be no Central Repository nor dependency management possible.

Whether you use Grade or Maven, both read the pom.xml Maven definition to build the dependency tree. It's at the core of the dependency dependencies system to pull all transitive dependencies and resolve version conflicts.

In other words, Maven can live without Gradle, but Gradle still needs Maven to exist. Maven just applies a standardized build based on the pom.xml while Gradle builds in in some way and generates a pom.xml as a build step if you want to actually publish your library.

Maven isn't perfect either

Now, I bashed a lot about Gradle, but Maven isn't perfect either. It has issues too. Their website sucks IMHO, it could welcome YAML as more compact alternative format, some plugins should be built-in and the format itself could be tweaked here and there. But overall I find it OK considering it's a format that lived more than 20 years.

The other drawback is a lack of flexibility. It's indeed rigid in how it expects your project to be and may become problematic if you need for example to mix multiple different techs. For example a building a node project, running a python script, etc. as part of the build procedure to place some extra stuff inside the produced artifact. But for that IMHO, it's better to use CI scripts, running as GitHub actions or GitLab pipelines to build a "mixed bundle". Let each tech stack build its own artifacts and combine them later through scripting. I favor that approach over pushing the build scripts customizations too far.

Take it with a grain of salt

While I bashed at Gradle and praised Maven, it should be taken with a grain of salt. At the end of the day, they are just tool and either can be used wisely or like a fool.

With maven too, you can also produce "monster pom.xml files" by using tons of plugins and super-complex configurations overriding all defaults. Likewise, Gradle is not necessarily a monster. Use it wisely, keep your build script clean, refrain from adding custom build steps and you will do just fine. It's not bad per-se.

It's just that by default, in the hands of average developers, Maven's pom.xml will tend to remain understandable (because it takes effort to escape out of the conventions) while Gradle's build.gradle will tend to become more complex and customized over time (because it's so easy to do so). All the small shortcuts now and little extra steps that stray away from the build conventions tend to become liabilities in the long term.

As said previously, Gradle's great flexibility and customizability of the build is both a boon and a curse. Although I prefer to build "generic" projects where I can, because it's by far simpler to maintain in the long run, using Gradle definitely has its place when you need more specific stuff that requires customization.

TL;DR: as a rule of thumb, Maven's pom.xmtends to remain fairly generic with time, while Gradle's build.gardle leans towards being highly customized and therefore complex. This is due to their "nature", while Maven is based on a rigid project "definition", Gradle is a free form build "script".

As a follow-up from an earlier article regarding the update to randomly generated default avatars for Passwordless.ID, I wanted to post a "how to". This is a beginner tutorial since making such avatars is actually really simple.

TL;DR; Here is the full demo.

The image format

The first thing you should think about is the image format, usually, one of:

• Jpeg: great for real user photos due to the high compression ratio. However, this compression also produces some "blur" on lines and sharp edges. As such it is not ideal for the avatars we are going to make.

• PNG: theses have lossless compression. In other words, every pixel remains exactly the same as it was originally drawn. Edges and lines remain "sharp".

• SVG: these are scalable vector graphics. Unlike a "raster of pixels", it is a declarative format describing shapes and paths.

Of course, you could also save it as a "100% quality" Jpeg to avoid any quality loss, but then it is larger than PNGs. Jpeg compression is amazing though for common photos.

In our case, we picked SVG for the upcoming avatar pictures. In the past, SVG was kind of avoided because support was not always well supported for all software platforms. This is however largely in the past.

SVG offers several benefits: the first is being scalable. Due to its vector nature, it is perfectly sharp at any scale, even if you zoom in on a 4K display. Other "raster" formats like Jpeg or PNG become "pixelated" when zooming in. The other is being more compact. While the byte size of Jpeg/PNG grows with picture size, SVG grows proportional to the shape's complexity. For relatively simple stuff like the avatars here, they are super compact.

The SVG "template"

SVG is an XML format that describes the shapes. As such, what will be generated is a big XML string. To be more exact, we will fill the template below with the appropriate values.

  <svg xmlns="http://www.w3.org/2000/svg" width="100" height="100">
    <defs>
      <linearGradient id="gradient" x1="${startX}" y1="${startY}" x2="${endX}" y2="${endY}">
        <stop offset="10%" stop-color="hsl(${startHue}, 100%, 50%)" />
        <stop offset="90%" stop-color="hsl(${endHue}, 100%, 50%)" />
      </linearGradient>
    </defs>
    <rect x="0" y="0" width="100" height="100" fill="url(#gradient)" />
    <text x="50" y="55" text-anchor="middle" dominant-baseline="middle" font-size="75" font-family="Times New Roman" fill="#ffffff">${char}</text>
  </svg>

Once this template is filled with meaningful values, you will obtain an avatar SVG image that can be stored as a plain normal ".svg" file.

Alternatively, you also deliver it as "data URL" since it is quite compact. This simply means encoding the resource directly instead of a "plain URL" fetching it. It is composed of two parts: the mime-type (image/svg+xml in this case) and the (base64 encoded) data. This can be used like any other URL in the src tag of an image as follows.

<img src="data:image/svg+xml;base64,{{the-base64-encoded-svg}}" />

Voilà, you got your image!

Getting some random values

The missing step is now filling this SVG template with some random values. Alternatively, if you want something more deterministic, you could also use the hash value of the name for example.

As you saw in the SVG template, instead of using RGB colors, HSL colors were used. This stands for Hue-Saturation-Lightness. This makes it easy to generate bright colors from all rainbow colors, with maximal color saturation and average "lightness".

  // Gradient colors
  const startHue = Math.round(Math.random() * 360);
  const endHue   = Math.round(Math.random() * 360);

  // Gradient direction
  const angle = Math.random() * 2 * Math.PI

  // Calculate the start and end points of the gradient
  const startX = (Math.cos(angle) + 1) / 2;
  const startY = (Math.sin(angle) + 1) / 2;
  const endX = 1 - startX;
  const endY = 1 - startY;

  // The character to appear on the avatar
  const char = name.charAt(0).toUpperCase()

For the gradient direction, it's a bit more tricky since an angle cannot be provided directly. There are some "transforms" available, but to ensure the widest compatibility with SVG renderers, sticking to the basics seems a safe bet. As such, the angle is converted to starting and ending coordinates for the gradient.

Thank you

The resulting full source code can be seen in the example provided at the beginning. :)

A Pen by Arnaud Dagnelies (codepen.io)

Whether it's good ol' servers, docker containers or lambda functions in the cloud, the code must run on physical machines in the end.

Nevertheless, their natures are very different, with both advantages and drawbacks. So let's cover the basics and quickly summarize the ups and downs of each.

Good ol' servers

Traditional servers should not be underestimated. By servers, we mean either physical hardware or cloud VMs. Even on cheap hardware or VMs, serving up to millions of requests per minute is commonplace. That's already a lot of users. Moreover, if the "stack" is local, with database and application logic on the same machine, there is no communication overhead which boosts performance to the maximum. It is a very cost-effective solution, but on the other hand, they require a bit more effort to administrate and require adequate Linux know-how.

Moreover, achieving redundancy and rolling updates requires attention. This is commonly solved by having at least two servers behind a load balancer to avoid any downtime. Regarding software updates, the usual way is to stop the application, update it then restart it, and perform this sequentially on the servers. The load balancer will take care of avoiding the temporarily unresponsive servers. While this is typically done manually in the beginning, it can also be automated with the right tools later on when the need arises, as well as scaling up to more servers.

Docker images

These images are run in containers that are typically distributed over larger distributed infrastructures. Containers also have the notion of being "volatile": they are typically quickly started and destroyed. For example, if the software crashes or isn't able to run properly, it can simply be destroyed and a new one created on the fly with the same image, effectively restoring it to its original state in a short time.

Using docker in production involves two levels: the software "images" and the "orchestrator" responsible to run these images on various servers. The latter, like Kubernetes, is typically offered as a service. With docker, scalability is achieved by simply launching "one more container" on the underlying infrastructure.

In the typical docker "stacks" the databases, APIs and UIs are run in separate containers. While this is not an ironclad rule, it is regarded as best practice to do so. This makes it possible to update and scale each part independently according to usage, on the other hand, this adds some overhead for network communication between containers. Some effort is also required to configure how all containers are distributed and "linked together" to interact properly.

The usual way updates are done is by simply launching new containers running the new image and then removing the obsolete ones afterwards. This is straightforward if images are independent, but if an update requires some coordinated update of multiple images at once, it might become more tricky.

Serverless

The idea here is that you only write the code and that it is invoked "on demand". You don't take care of the infrastructure at all; the provider will take care of it. This comfort has a strong constraint: it must be completely stateless. There is no persistence guarantee between requests since it can run "anywhere". This requires planning accordingly in advance, usually coupling it with databases as a service and "object storage" services to persist files.

This "serverless" paradigm comes in two flavours, related to their underlying technology.

Like AWS Lambda

This launches a minimal docker container, processes the request, reuses it for further requests and destroys it after some idle time. This offers large freedom regarding the programming language and frameworks to process the request and almost no constraints regarding execution. However, they do have their downside. In particular, the very first request will take a long time, also called a "cold start" since it involves starting an environment. For the right price, it is possible to leave them "hot".

If you leave them hot, they are expensive, if not, your users often suffer from high latencies. As such, they are best suited for on-demand background jobs or long-running tasks IMHO.

Like Cloudflare Workers

This runs JS/TS functions directly in a "chrome-based engine" to process the function. This is cheap, fast and can be run in a distributed manner easily. This is perfect for small functions running "at the edge" of the network. Every time a request is processed, the function's code is loaded and interpreted on the fly to produce the response. Async code invoking other services is also possible.

These functions should be as self-contained as possible, and providers usually put clear constraints on the overall size and max processing time. While it is great for small functions, it becomes counter-productive for larger programs offering complex functionality.

What should I pick?

Scaling can be achieved with either and for most things there is no clear-cut choice which one is clearly better. It's rather a balancing of advantages and disadvantages, mixed in with the team's expertise, experience and preferences.

What follows are not iron-clad rules but rather rule-of-thumb pieces of advice to be in the comfort zone.

Like I said, the smileys are not a hard requirement. For example, you can also have simple code on servers and complex code on serverless too, it's just likelier that you will likely leave the comfort zone and lose some of its benefits.

If the functionality is simple, go serverless with Cloudflare Workers like offerings! This is free to start with using most providers, scales well, offers top latencies worldwide and has excellent pricing even on large scale. The ecosystem might not be very mature, but with simple functionality comes low complexity where an ecosystem is not that critical.

If you require large amounts of memory or GPUs, physical servers or VMs might be advantageous because of pricing alone and its efficiency since there are no abstractions in between.

If you have a complex software landscape with many web services, use docker. Since the web services can easily be packaged as docker images and almost every open source software already has docker images too, it is simplest to use them directly.

If it's something like background jobs running once in a while, use AWS Lambda-like functions. They can process complex stuff, run longer and you are not worried about cold starts and latency issues.

Some use cases

Passwordless.ID

Here, the API could be divided into individual, largely decoupled functions of reasonable complexity. That made Cloudflare Workers a perfect candidate. Thanks to that, the code can run over worldwide distributed datacentres with optimal latency. Perfect for manageability, scalability and pricing.

KeyValue.Rocks

This is a data-hungry beast. It stores large amounts of data in memory for fast access and high throughput. As such, VMs were used. They are cheaper and less complex than orchestrating a docker deployment. Since the project was kind of low activity with rare updates meanwhile, it was adequate.

Because there is no worldwide quality source for addresses!

Really, that's no joke. There are many commercial providers for "industrialized countries" of variable quality/pricing but worldwide coverage is lacking, the data formats are diverse and the license terms provider specific.

There are also some open source projects related to addresses though, each with their gochas. Two of these projects are mentioned in the last section "Honorable Mentions" at the end of this article, along with their drawbacks.

This project is born in order to provide quality addresses with worldwide coverage under an open license, by directly extracting addresses from the raw data dumps of OpenStreetMap.

Birth of OpenStreetData.org

How does the it look like? Here is a screenshot, but if you prefer, check out the website directly.

It is divided into two parts: extracts and addresses. Another "points of interest" was planned, but not further developed due to lack of time.

Country dumps

Country extracts are provided in two formats:

• PBF, the native OpenStreetMap binary format. This format is very compact and many tools can handle it efficiently. Nevertheless, it is not always very practical to handle due to its low-level nature. It's basically a huge list of points with IDs, lines that reference these IDs and relations that reference the lines.

• GeoJson sequences. It's a text file where each line is a "feature": a JSON object with arbitrary properties and a geometry with coordinates. Although the file size is typically larger and the processing sometimes slower, it offers other benefits. The JSON format is universal, the line-based sequence makes it straightforward to filter it with grep-like tools and the geometry can be parsed directly without requiring to go through the whole file.

Note though that both formats are not 100% equivalent. During the conversion process, some choices were required to be made. In particular, in the original PBF a "closed line" (where the last point is the same as the first) could be interpreted as a line or as an area either way. There is no clear-cut indication whether it's a "line" happening to turn in a circle, like a roundabout, or a polygonal "area", like a building outline. This led to "closed lines" being interpreted as lines or polygons based on a lot of hand-picked feature properties. For example, if building=... was part of the properties, it was considered a polygon, except if an area=false tag was also present, and so on.

Administrative areas

Despite not being shown on the site, extracting precise boundaries of a country's provinces, regions, counties, cities, suburbs and so on was the first crucial step. How a country is subdivided into smaller areas varies greatly from country to country and is abstracted under the name "administrative areas" of various levels.

This step is crucial because of the way addresses are extracted. Streets and houses were extracted using "spatial joins" with the administrative areas. Their coordinates were used to determine which administrative areas (city, county, province...) they belong to, as well as the postal code ...if postal code areas are defined in the country.

Currently, the reason of missing (or wrong) addresses for some countries are improper mapping of the administrative areas.

Streets

Here is an example of the "streets" for Austria:

168769 rows × 7 columns

It extracted all streets having a name from the raw data and determined the administrative areas and postal code it belongs according to their centroid. As such, it is a slightly simplified streets list. If a street might cross multiple cities or postal codes for example, it will solely be listed in the "main one" (according to its center). For more precise addresses, see below.

Note that "suburb" may be empty depending on the size of the city. This is normal since not all cities are further divided into suburbs.

Houses

Houses is a dataset listing each house (anything with a house number) individually, including its coordinates and the administrative areas it lies within.

Addresses

In this case, the houses are "merged" into streets with house numbers. Unlike the "streets" approach, it results in a more fine-grained dataset.

• it includes only streets with at least a single house (number)

• it differentiates between street sections with house number ranges belonging to different administrative areas or postal codes

• it differentiates between different sides of the street (with odd/even house numbers) belonging to different administrative areas or postal codes

• it has boundaries

Here is an example of such an address file for Austria.

146322 rows × 11 columns

It may not be perfect, for example, the first line with a misinterpreted city name is quite mysterious.

Challenges

"Big Data"

Dealing with large data is challenging. It's not thousands of points, it's not millions, it's many billions of points, lines, polygons and relations.

Seems like a detail? Well, for example, you cannot even load the planet's data at once in memory. It's simply too big.

You cannot just "do as you please" with inefficient code. Every line of code, every operation, must be crafted with care, well thought out, and fine-tuned to keep processing time and memory to a minimum.

As an example, just for processing the data of a single country, even 32Gb RAM is not enough for larger countries and it takes many hours with the current code, despite best efforts.

Producing precise country extracts

There are sites like geofabrik.de providing country extracts to download. However, they turned out to be not precise enough for me. They use "simplified country border polygons" that are "cutting corners" and therefore missing addresses in areas near the borders. So I had to "split the planet" myself.

To do so, the first step was to extract exact country boundaries. Interestingly, these might change over time. Usually, it's minor modifications like slightly adjusting the border or correcting mistakes. But sometimes the border might move a bit more in "unstable" parts of the world. The point here is that these borders are not "definitive" but evolve slightly over time.

The next step is splitting the world into country extracts. Here again, it cannot be naively done in a single step. Doing so, even 256Gb RAM would not suffice to split at once. So the splitting must be done in multiple steps: first in continents, then in regions, then in countries so that it fits in a "reasonable" amount of memory.

And cutting whole continents with a super precise boundary constituted from millions of points is not efficient either. On the other hand, computing the total bounds of the continent is pointless too. For example, the outer bounds of just France would cover almost the whole world since it possesses many islands around the world as part of its territory. You get the point, some extra work must be done to simplify the geometry without losing stuff but without including too much either.

Then, there are ways or area relations that cross boundaries. Some things from the raw data are not always clear whether it's a "closed line" or an "area", and so on. It's full of technical details which make even producing what look like simple "country extracts" challenging.

Heterogenous data

The OpenStreetMap raw data is not a homogenous clearly defined dataset. It is a huge amount of points, lines and relations, each with completely arbitrary properties. For example, a statue might be a point with metadata indicating when it was built, and from whom, along with some tourist guide number. Depending on where you look on the map, you may also notice different habits of mappers using a diversified arsenal of "tags" to describe things and the community as a whole has different opinions on how to do things, for example with addresses, which often have local flavours.

If you dig into the raw data of OpenStreetMap, you will find interesting things. For example, you will find tags like addr:street=... and addr:city=... which sounds promising. These are also very simple (and quick) to extract since it's attached directly to the data. Great right? Well, it would be this data was complete but it is far from. Depending on the country you are looking at, it might be mostly widespread or barely used. Even if it's there, the coverage and the content are usually quite fuzzy. For example, the street might be named "Wall Street" on one building while another building uses "Wall St.". Likewise, the city in one building may be "N.Y. City" while another uses "New York". Postcodes may also be written in individual houses, but not match the postal boundary accurately, etc. This makes processing these tags directly error-prone. It's better than nothing but there are ways to make it better.

Namely what we did. Spatial joins of houses/streets points into administrative/postal areas in order to extract the most information possible. If those areas are not mapped, a fallback to the tags is used, but only as "fallback" since they are usually not that precise.

Manual labor

Doing this is quite some work. It's not just running a process and be done with it. It's craftsmanship where you change a few lines of code and manually inspect the results. Just checking if more streets/houses/addresses are produced is not sufficient either. It might be that the output is of worse quality because street names are duplicated or listed in the wrong "areas" or some other data mistakes. It might also be that the "couple of lines change" works perfectly for one country but breaks in another because of local differences, like for example the presence or absence of postal codes.

Sometimes, you also see odd things in the data. When this happens you usually spend some time to investigate "why" it is so. Is it the raw map data that is strange? Is it some situation you did not think of? Is there a bug in the code? Is some third-party library not working as expected?... It's really full of weird things, from buildings on the map having mistakenly used "national boundaries" tags to sudden performance drops in third-party libraries when calling a certain function.

Addresses are crazy

Below, I will illustrate how addresses are crazy. It's not something that is homogenous worldwide. It's full of regional quirks.

Is it a country or not?

You may think that something as basic as countries and boundaries is clear-cut. But it is not. Take Kosovo for example.

For half of the world (marked in red), Kosovo constitutes a province of Serbia, while for the other half of the world (marked in blue) Kosovo is recognized as an independent country.

...and that's not unique to Kosovo. There are plenty of regions in the world where territory is disputed, where border shift with local wars and where sovereignty depends on who you ask.

What stance do I take here? I simply use the list of countries as defined by the united nations, defined by their country codes ISO 3166-1. It might not be ideal, but it is pragmatic.

A city with many borders

On a small scale level, borders can be crazy too. For example, check out the little town of Baarle-Nassau, located in the south of the Netherlands, near the Belgium border.

This town contains 22 small exclaves of the Belgian town Baarle-Hertog, some of which contain counter-exclaves of Nassau. The borders cross streets in the middle, sometimes multiple times and a single house might have a Belgium address on one side and a Netherlands address on the other. As you see, extracting addresses can quickly become challenging. ;)

A city center without street names

Not all addresses are based on "streets". Take a look a Mannheim for example. There, the city center is divided like a big grid.

There, each "block" has an identifier, like "C3" while the streets are unnamed. Likewise, house number does not belong to a "street" but to a block. In other words, your address might be "C3, 17" if you live in building 17 of block "C3".

Fancy house numbers

Do you want to use regexes to filter valid house numbers? Well, that might not really work out. For example, the following image is a valid Vietnamese house number, near the coasts of Ho Chi Minh city.

The world is full of surprises. Also regarding addresses, it's full of diversity and local quirks and I believe there is nothing that does not exist.

Honorable mentions

There are also two noticeable open source projects trying to bring addresses to the public domain.

openaddresses.io

This is probably the most famous one. It works by running various "scraping scripts" against various "raw data sources". The result of this approach has few drawbacks though, directly related to its approach.

• the licensing is problematic. Basically, it says "use this data according to the license from the data source" ...which is not obvious, since the original issue is not directly linked, often in native language and the licensing terms makes the usage of this data questionable.

• the coverage is lacking

• the scraping sometimes breaks or is outdated because of changes in the raw source

• despite being "open", some things are obfuscated and make reproducing or direct downloads difficult

osmnames.org

Despite less known, this is IMHO a better source of addresses. It is based on addresses extraction from OpenStreetMap and therefore has worldwide coverage and a homogenous license: the "ODbL - Open Database License".

The only drawback it has IMHO is:

• the lack of postal codes

• the admin_level mapping not ideal

The lack of postal codes may seem like a detail, but it is crucial for addresses. Without it, addresses are simply incomplete. Since this project is open source, I also tried to improve it by adding postal code (see issue) but it turned out too be too difficult/challenging for me. Mostly because I am unfamiliar with PostGIS. The code however, is of quality. This lead me to the current project.

The second issue is more subtle, and leads to missing addresses in some countries because administrative boundaries are not properly mapped. Also, the code is not suited for experimentation and from my understanding, there were ways to "get more" out of the raw OpenStreetMap data dumps than how they did it.