About Myself

An enthusiast software engineer

Passionate about languages and compilers

Worked in the V8 team in Google

Overall, worked on JIT compilers for +7 years

Now working on scalable, fault tolerant,
web facing distributed systems...

...serving vitual reality on the web!

Javascript and me

I started as a Javascript hater

When I sow Javascript for the 1st time (in 1995) I vowed that I would never touch such an abomination, even with a 10 feet pole

Eventually, I changed my mind, but...

Let's change subject!

What are we talking about?

Build Tools!

Why?

I tried several, and
I was not happy...

What's the problem?

Simple is good

Everybody starts with a few scripts

The project grows, the scripts get slow...

Dependency tracking is the needed feature

(performing build steps in parallel also helps)

The recipe for speed

independency
(A, B, C, need A, skip B and C)

dependency
(A ► B ► C, need C, skip A and B)

parallelism
(A, B, C, need all, run in parallel)

State of the build

Grunt

just a task runner

very verbose

build files get huge quickly

State of the build

Gulp

A bit more pleasant

Streams are interesting

If you have multi-file operations you end up fighting it badly

Why do I have to write code in a specific language?

State of the build

Broccoli

Brilliant concept

So-and-so execution

sometimes all those temporary trees get in your way

Why do I have to write code in a specific language?

State of the build

Other tools

Webpack (specific use, fast only when hot)

Jake (mostly like make)

Gradle (you must know Groovy)

Blaze (more about this later)

Why do I have to write code in a specific language?

Wishlist

Builds need to be...

deterministic

reliable

fast

Traditionally...

Builds happen in a build environment

The environment is a file tree

Each build step mutates the file tree

How do you reliably detect if the build environment contains the proper result of a build step?

You don't...

Dependency tracking howto

File system based

You use the file system and check the mtime of input and output files
(Jake, Make...)

(can get cumbersome)

Dependency tracking howto

Internally tracked

The build system has an internal graph that carries the build artifacts
(Gulp, WebPack...)

Fast when the build system stays running
(like while watching)

A better way

Track contents and not mtime!

Just like Git does

...and just like Bazel does...

An even better way

Store everything in a content addressable file system

(Git is a content addressable database)

Use this file system as environment for build steps

One step further

Isolate each build step in a container

Expose the inputs as a read only tree

Capture the output and snapshot it in the content addressable storage

Cache every computation!

The FFTT Choice

Fast

Correct

Pick two!

(just like Bazel)

Build Systems

meet
-

Functional Programming

Functional Programming

Immutable, copy on write data structures

(with structural sharing)

Memoization of function results

Code or Configuration?

Both, but each where appropriate

The build graph is declarative and functional

Each build step is imperative but confined inside its container

Programming language independence!

One more step further...

-

Build Systems

meet
-

Continuous Integration

A CI system is a build system...

...with its own configuration language!

The dependency graph description is usually too verbose to be practical

Tipically only one or two developers in a team know how to work on it

Nobody wants to learn one more build system

A modular CI system

FFTT performs builds

Something else detects changes in the source and invokes builds

This "something else" might watch a remote git repository...

...builds could trigger deployments...

Add a web view of the content addressable store...

Fast deployments

Why docker builds are slow

Use the FFTT repo as a read only volume

Deploy containers mounting that volume

(let them access that volume over NFS or a clustering file system like GlusterFS)

No need to wait for a rebuild, ever!

Parallel, distributed builds

A side effect of the pure functional nature of FFTT

Actually, the simplest way to implement it is naturally parallel!

Using docker swarm (or maybe kubernetes) as container runner the builds would be distributed

Under the hood

The repo, and the linking issue

The exposed file system and fuse

The repo, as it could be

How Javascript has been used

Todo list

Atomic operations on the repo

Garbage collection

Fuse view of the repo

Explicit tagging

Repo transfer network protocol

Repo web browser

Take Over the World!

Related Work

git

ostree

docker

bazel

Wrapping up

Even if you will not use FFTT, I hope you learned something about build tools

Maybe you'll all switch to Bazel...

If you did like it, please plan to contribute!