How It Started

Back in 2020, I tweeted about the project:

Can’t see the embed? View the tweet on X →

At the time, I was excited. I had this vision of a comprehensive guide that would take someone from zero to building a real compiler in Rust. Not a toy language, but something with a proper parser, type system, and code generation.

The early version got some attention, people seemed interested, and I thought: “Great! I’ll wrap this up in a few months.”

Six years later, here we are.

What Took So Long?

Honestly? Everything.

Life happened. Losses. Jobs changed. Priorities shifted. I’m a father now, and that’s a lot of responsibility.

The scope kept growing. Every time I thought I was done, I’d realize there was another important concept I hadn’t covered. Lifetime semantics. Borrow checking. LLVM integration. Optimization passes. The list kept expanding.

I didn’t want to write a typical compiler book. You know the ones. 1000 pages about parser theory that put you to sleep by chapter 3. I wanted something practical, something you could actually follow and build with. That meant finding the right balance between theory and practice, which is harder than it sounds.

But you know what? All of that is okay. Because the book that exists now is so much better than what I envisioned in 2020.

What’s Actually In It

The book walks through building a complete programming language from scratch:

Part I: Foundations - Lexical analysis and tokenization - Parsing with recursive descent and Pratt parsing - Abstract syntax trees and intermediate representations

Part II: Type Systems - Type checking and inference - Polymorphism and generics - Lifetimes and ownership (the Rust way)

Part III: Code Generation - LLVM basics and IR generation - Optimization passes - Linking and producing executables

Part IV: Advanced Topics - Garbage collection and memory management - Concurrency primitives - Interop with C and other languages

Every chapter has complete, working code. Not pseudo-code, not simplified examples. Real Rust code that compiles and runs. The full implementation is on GitHub, and it’s a legitimate compiler for a small but complete language.

What I Learned

Writing this book taught me more than I expected.

The biggest lesson is that writing forces clarity. When you have to explain something to another person, you find out fast what you actually understand versus what you only thought you understood. I learned Rust more deeply by teaching it.

I also relearned the value of small, steady progress. Some weeks I only wrote 200 words, and that’s fine; 200 words a week is 10,000 words a year. And at some point you have to accept that done beats perfect. There are probably typos in the book, some explanations could be clearer, a few examples could be more elegant. But if I’d waited for perfect, I’d never have released it. What kept me going through all of that were the people who read early drafts, opened issues on GitHub, and asked questions on Discord. Every bit of that feedback made the book better.

Why Rust?

Rust’s safety guarantees, performance, and modern tooling made it the obvious choice. But honestly, the bigger reason I kept at this was that I really don’t like having a half-baked project hanging over my head. You know the feeling? That nagging sense of something unfinished, something you started with excitement that’s just sitting there incomplete.

Six years is a long time to carry that around. But now it’s done, and that feels good.

What’s Next?

The book is done, but the project isn’t over.

I want to add more examples. Maybe a chapter on implementing a small standard library. Perhaps a section on JIT compilation. There’s always more to explore.

I’m also planning a series of blog posts diving deeper into specific topics from the book. Type inference algorithms. Register allocation. Optimization strategies. All the stuff that didn’t quite fit in the main narrative.

And honestly? I’m just excited to see what people build with it. The whole point was to lower the barrier to creating languages. If even a few people use this to build something cool, that’s worth six years.

A Note on Mojo🔥

Speaking of creating languages, I should mention that I’m now part of the team at Modular. Mojo is a new programming language for AI that combines Python’s usability with C-level performance. It’s a serious language design effort, and if you’re interested in systems programming, high-performance computing, or just how modern languages are built, check out the Mojo documentation.

Honestly? If Mojo existed when I started this book, that would have been my choice. It’s much more powerful than Rust for the kind of low-level systems work compilers need, but also simpler. That’s the sweet spot. a language that gives you more power without the complexity overhead. But that’s the thing about language design, timing matters. In 2020, Rust was the right tool. Now we have even better options.

We’re on the path to Mojo 1.0, and it’s been incredible to see how the language has matured. The experience of writing this book definitely influenced how I think about language design. And working on Mojo has deepened my appreciation for the tradeoffs and decisions that go into creating a production language. It’s one thing to teach compiler concepts, another thing entirely to build a language that people will actually use in production.

Final Thoughts

Six years feels like a long time. But looking back, it doesn’t feel wasted. Every chapter I struggled with, every example I rewrote, every concept I had to learn properly, it all contributed to something I’m genuinely proud of.

If you’re working on something that’s taking longer than expected, whether it’s a book, a project, or just learning something new, that’s okay. Progress isn’t always linear. Some things just take time.

And when you’re done, whenever that is, it’ll feel worth it.

The book is live at createlang.rs. The code is on GitHub. And if you have questions, find bugs, or just want to chat about compilers or language design, open an issue on the GitHub repository.

Thanks for following along on this journey. Here’s to the next one.

Happy New Year. 🎉

Acknowledgments

Huge thanks to Chris Lattner and Abdul Dakkak for taking the time to review drafts of this paper. Your feedback was incredibly helpful and pushed me to clarify and strengthen the ideas. I’m really grateful for your encouragement and thoughtful comments.

How This Started

So this whole thing started back in 2019 with this tweet thread about using category theory to understand Rust’s lifetime system. I was thinking about lifetimes as categories, references as functors, that kind of thing.

I was working with Rust a lot and thinking about borrow checking and lifetime systems, and the categorical structure felt important. Also the origin goes back to my earlier blog on Variance in Rust. But later I started wondering: what if we could geometrize this? What if undefined behavior isn’t just a type system violation, but has actual geometric structure? Like, what if memory safety violations are holes in some space of program states?

Fast forward to 2025, and I started playing around with the idea more seriously. I was using ChatGPT 5.1 to bounce ideas around, and it made a suggestion that ended up being crucial: use the Alexandroff topology on a poset of abstract program states. That insight was the key that made everything click. Once I had that, the rest of the framework started falling into place.

The Basic Idea

Here’s the pitch in plain English:

Systems languages like Rust, C++, and Mojo try to give you low-level control while keeping you safe from things like use-after-free, iterator invalidation, and all the other ways you can shoot yourself in the foot with memory.

The standard way to think about this is through type systems (Rust’s borrow checker), logical models (RustBelt), or operational semantics (Stacked Borrows). These are great, but they’re often language-specific and pretty complex.

My approach is different. I’m treating program states as points in a geometric space, and undefined behavior as topological holes in that space. Specifically:

1. States form a partially ordered set (poset)

Think of each state as representing what your program knows about memory at some point. If state w can safely evolve into state w', we say w ≤ w'. So the “safe extension” relation gives you a poset.

2. The Alexandroff topology

This was the key insight GPT 5.1 gave me. In the Alexandroff topology, a set is “open” if it’s upward-closed: if a state is in the set, all safe extensions of it are also in the set. This perfectly captures the idea that safety is monotonic. Once you’re in a safe state, staying safe means continuing upward.

3. Build a simplicial complex

From the poset, you can build a simplicial complex. Vertices are states, edges are safe transitions, 2-simplices are pairs of operations that commute safely, and so on. This complex describes the shape of the safe state space.

4. Homology finds the holes

This is the part that hooked me. If there’s a cycle in your state space that you can’t fill in (like trying to use a reference after it’s been invalidated), that shows up as nontrivial homology. Specifically, it’s a loop that doesn’t bound a 2-dimensional disk, which means you have a hole in (the first homology group).

Undefined behavior is those holes.

A Concrete Example

Let’s do Rust. Consider this pattern (which thankfully doesn’t compile):

let mut v = vec![1, 2, 3];
let r = &v[0];      // borrow v
v.push(4);          // realloc may move v (UB if r is used)
println!("{}", *r); // use-after-free

In the paper, I model this as a state space with four states:

• w₀: v exists
• w₁: reference r to v[0] is created
• w₂: v gets reallocated (invalidates r)
• w₃: r is dereferenced (UB!)

The problem is that there’s a cycle w₁ → w₂ (reallocate while borrowed) and w₂ → w₁ (somehow un-reallocate?) that doesn’t bound a 2-simplex. There’s no safe “commuting square” where you can do both operations in either order. This cycle is a generator of , a 1-dimensional hole.

Rust’s borrow checker prevents this statically by not letting you do v.push(4) while r is live. But in C++ or unsafe Rust, the hole is there. And you can detect it by computing homology.

Wait, Lifetimes Are a Category?

Yeah, this part is kind of fun. In addition to the topological view, the paper gives a categorical semantics for Rust’s lifetime system:

• Lifetimes form a category where objects are lifetime parameters ('a, 'b, etc.) and morphisms are outlives relations ('a: 'b).
• Reference types are bifunctors from (Lifetimes × Types) to Types. So &'a T is functorial in both the lifetime 'a and the type T.
• Reborrows are natural transformations. When you reborrow &'a T to &'b T with 'a: 'b, that’s a natural transformation between bifunctors.

This is pretty standard category theory (in the sense of Mac Lane’s Categories for the Working Mathematician), but I think it’s cool that Rust’s lifetime system actually is a functorial construction. It’s not just an analogy.

C++ and Mojo Too

The framework isn’t Rust-specific. In the paper I also work through:

• C++: I build a toy example with raw pointers and iterator invalidation happening in the same function. The complex has , meaning there are two independent UB holes. You can trigger one, the other, or both.

• Mojo: Mojo has this cool system of origins that parameterize lifetimes. I sketch how origins fit into the same geometric picture, and how you can use cohomology (specifically, sheaf cohomology on the poset of origins) to detect global inconsistencies.

The details are in the paper. The point is that once you have the geometric layer, you can apply it to any language where you can define abstract states and safe transitions.

Can You Actually Compute This?

Short answer: yes, on small scopes.

Homology is just linear algebra. If you have abstract states and transitions, you build a simplicial complex with vertices and at most edges, and computing reduces to finding the kernel and image of some matrices over . For intraprocedural analysis (like checking a single function or loop), this is totally doable.

The paper includes a high-level algorithm:

1. Extract abstract states and safe transitions from your program (using existing static analysis).
2. Build the simplicial complex.
3. Compute .
4. Report any nontrivial cycles as candidate UB patterns.

This isn’t a replacement for the borrow checker or sanitizers. It’s a complementary technique. You’d run it on suspicious regions (like around unsafe blocks) to get a geometric sanity check.

Why I Think This Matters

For language designers, thinking about UB geometrically gives you design principles. If you can design a type system so that certain complexes are contractible (no holes), you’ve ruled out whole classes of bugs by construction. For static analysis, you get algebraic invariants (the homology groups) that are language-independent, so any tool that can already extract abstract states can plug into the framework.

And for the rest of us, it’s a mental model. When you write unsafe code, you’re navigating a space that has holes in it, and the borrow checker is trying to keep you in the safe, contractible regions. Knowing that is half the battle when you’re reasoning about what your unsafe block is really doing.

What’s Next?

I want to build a prototype. Take Rust MIR or some C++ intermediate representation, extract small state complexes around uses of unsafe or raw pointers, compute , and see if it finds known bugs. If it does, that’s validation. If it finds new ones, that’s even better.

There are also some deeper directions:

• Concurrency: Extend the poset to include thread interleavings. Use directed topology or directed homology to detect data races as higher-dimensional holes.
• Type system integration: Use homological properties as meta-invariants. Design type rules that guarantee contractibility.
• Sheaf cohomology: Treat types and invariants as sheaves over the state poset and use cohomology to find global inconsistencies.

But honestly, I’m just excited that the basic idea works. It’s been six years since that tweet, and turning a vague intuition into actual math felt really satisfying.

Final Thoughts

From a random tweet in 2019 to a finished paper in 2025, this has been a weird journey. The idea that you can think about memory safety bugs as literal holes in a topological space still feels kind of magical to me.

A Note on Publication

I tried to submit this to arXiv, but couldn’t get an endorsement. My old grad school advisors and colleagues aren’t eligible due to arXiv’s strict policies on endorser recency and paper count. The endorsement system makes sense for preventing spam, but it also creates a barrier for researchers who aren’t currently in academia or don’t have recent connections to active arXiv users. It’s a bit of a catch-22: you need endorsement to build a publication record, but you need a publication record (or the right connections) to get endorsed.

So for now, the paper lives here. The web is a perfectly fine place for ideas to exist.

Thanks for reading.

Why Mojo?

A few reasons. The syntax is close to Python but the performance is closer to C, and the type system catches a lot of the mistakes you’d normally only find at runtime. The bigger draw for me is portability: the same Mojo code runs on NVIDIA, AMD, and Apple GPUs (Apple support is still partial), so you’re not rewriting kernels for every vendor. You also get direct access to GPU intrinsics when you need them, without dropping all the way down to CUDA.

What you’ll learn

The puzzles go in order. You start with thread indexing and work up to tensor cores, and along the way you end up understanding both why GPU code is shaped the way it is and how to write it. All 34 are live now.

Start with puzzle 1: puzzles.modular.com.

Infrastructure from Code

In the last couple of years, there has been a new trend aimed at simplifying code deployment in cloud. If you’ve likely seen drawbacks withInfrastructure-as-Code (IaC)tools such as Terraform, Pulumi, AWS CloudFormation, etc., you’re not alone.Each tool has its own terminology, often in the form of a DSL, library/package, or YAML extension 🤕Decoupled from the application code, meaning that using resources in code requires separately creating the necessary cloud resources

For certain applications and larger companies, these requirements might be manageable. However, for smaller companies and individual developers, they can be overwhelming.

So, the main idea ofInfrastructure-from-Code (IfC)is to simplify IaC by being as close to the application as possible. This means introducing little to no new syntax and providing a way to create and manage cloud resources needed for deploying the application as conveniently as possible, for example, with a CLI.

In fact,Modal is an IfC solution for the computing stack, especially for ML/DL apps.

If you want to know more about IfC, have a look at my awesome-infrastructure-from-code resources.

Modal’s Pitch

Modal’s goal is tomake running code in the cloudfeel likeyou’re running code locally. You don’t need to run any commands to rebuild, push containers, or go to a web UI to download logs.

Installation and Official Guide

First, make sure you’re signed in modal.com. Then the installation guide

pip install modal
python3 -m modal setup

installs the modal package which comes with its own CLI. Also configures the CLI and sets up the API token once. After that, let’s create a dev environment for our deep dive

modal environment create dev
modal config set-environment dev

Note: All codes are available in my GitHub.

Runtime Spec via stub

Let’s see what that means. First,

git clone https://github.com/ehsanmok/blog && cd blog/code/modal-deep-dive

and have a look at lib/empty.py

import modal
stub = modal.Stub("empty")

Let’s deploy an empty stub via

modal deploy lib/empty.py

(since I’m using Poetry, I factor out the poetry run part).

and the Modal dashboard view

Ok! so far so good. We just deployed an empty stub. You can think about it like acontainer(either in abstract sense or like docker container) and here the container is empty! We will expand on that later.

Let’s continue with lib/hello.py

import modal

stub = modal.Stub("hello")

@stub.function()
def hello():
    print("hello, world!")

Let’s take a look at Stub description (emphasis is mine)

A Stub is adescriptionof how to create a Modal application. The stub object principally describes Modal objects (Function, Image, Secret, etc.) associated with the application. It hasthree responsibilities:>

Syncing of identities across processes (your local Python interpreter and every Modal worker active in your application).>Making Objects stay alive and not be garbage collected for as long as the app lives (see App lifetime below).>*Manage log collection for everything that happens inside your code.See what happens if we deploy our hello world via

modal deploy lib/hello.py

and dashboard says it’s deployed but is “Currently idle”, because it’s not receiving any input

in fact in order to execute the stubbed hello function we should do modal run lib/hello.py.

From the output, we can see it

1. Created a mount of some sort (will expand on that later)
2. Created our application hello
3. Ran the app
4. Finally stopped the app

Let’s Dive Deep

What happens when we do modal run?

The official Modal document describes it as

How does it work?

Modal takes your code, puts it in a container, and executes it in the cloud.

Where does it run? Modal runs it in its own cloud environment. The benefit is that we solve all the hard infrastructure problems for you, so you don’t have to do anything. You don’t need to mess with Kubernetes, Docker or even an AWS account.

Official Modal doc

In the outset, Modal intelligentlywraps our code and all of its dependencies (we will get to it later) with metadatacontainerizes*and finally runs / deploys it

A few important points:the containerization isNOTvia docker. Modal has created its ownOCI compatible container servicesuitable for ML apps including heavy duty ones. An issue with the default docker container is being slow when for example including large ML model. Modal has its own container runner (compatible with runc and gvisor) and image builder.Modal has created it’s own filesystem (in Rust 🦀)that maps everything into a performant container via its FS. This innovating approach makes creating and deploying large containers fast and scalable.

We decided to not build this on top of tools like Docker/Kubernetes because we want infrastructure to befast… Modal has no problem building a 100GB container, and then booting up 100 of those containers — you can do the whole thing in a few seconds. This is what it’s built for.

Eric Bernhardsson’s (Modal CEO) blog

In fact, the essence of docker is chroot and Unix FileSystem. So with their performant custom FS, Modal has successfully created a veryperformant runtime for cloud.

To have a better idea of what is under-the-hood, we use modal shell

modal shell lib/hello.py
✓ Initialized. View run at https://modal.com/apps/ap-t46rTYWQKvHzcvjJ2uvpGr
✓ Created objects.
├── 🔨 Created mount /Users/ehsan/workspace/blog/modal-deep-dive/lib
└── 🔨 Created hello.
Spawning /bin/bash
root@modal:~#
root@modal:~# ls
intro
root@modal:~# cd lib/
root@modal:~/lib# ls
__init__.py  empty.py  hello.py

Interesting! modal shell spawns bash and shows where our python code is placed. Let’s see what is at the root directory

root@modal:~# cd /
root@modal:/# ls
bin  boot  dev  dummy_plug  etc  home  lib  lib64  media  mnt  modal_requirements.txt  opt  pkg  proc  root  run  sbin  srv  sys  tmp  usr  var
root@modal:/# cat modal_requirements.txt
# Pins Modal dependencies installed within the container runtime.
aiohttp==3.8.3
aiostream==0.4.4
asgiref==3.5.2
certifi>=2022.12.07
cloudpickle==2.0.0;python_version<'3.11'
cloudpickle==2.2.0;python_version>='3.11'
ddtrace==1.5.2;python_version<'3.11'
fastapi==0.88.0
fastprogress==1.0.0
grpclib==0.4.3
importlib_metadata==4.8.1
ipython>=7.34.0
protobuf>=3.19.0
python-multipart>=0.0.5
rich==12.3.0
tblib==1.7.0
toml==0.10.2
typer==0.6.1
types-certifi==2021.10.8.3
types-toml==0.10.4
typeguard>=3.0.0

So it looks like a usual Unix FS with a few new addops like modal_requirements.txt where we can see its content.

root@modal:/# python
Python 3.10.8 (main, Dec  6 2022, 14:24:03) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cloudpickle
>>>
>>> class Unpickleable:
...     def __reduce__(self):
...         raise TypeError("This object cannot be pickled")
...
>>> obj = Unpickleable()
>>>
>>> try:
...     serialized_obj = cloudpickle.dumps(obj)
... except Exception as e:
...     print(f"Error: {e}")
...
Error: This object cannot be pickled

message Function {
  string module_name = 1;
  string function_name = 2;
  repeated string mount_ids = 3;
  string image_id = 4;
  bytes function_serialized = 6;

  enum DefinitionType {
    DEFINITION_TYPE_UNSPECIFIED = 0;
    DEFINITION_TYPE_SERIALIZED = 1;
    DEFINITION_TYPE_FILE = 2;
  }
  DefinitionType definition_type = 7;

  enum FunctionType {
    FUNCTION_TYPE_UNSPECIFIED = 0;
    FUNCTION_TYPE_GENERATOR = 1;
    FUNCTION_TYPE_FUNCTION = 2;
  }
  FunctionType function_type = 8;

  Resources resources = 9;
  repeated string secret_ids = 10;

  RateLimit rate_limit = 11;
  WebhookConfig webhook_config = 15;

  repeated SharedVolumeMount shared_volume_mounts = 16;

  optional string proxy_id = 17;

  FunctionRetryPolicy retry_policy = 18;

  uint32 concurrency_limit = 19;

  bool keep_warm = 20;

  uint32 timeout_secs = 21;

  PTYInfo pty_info = 22;
  bytes class_serialized = 23;

  uint32 task_idle_timeout_secs = 25;

  CloudProvider cloud_provider = 26;

  uint32 warm_pool_size = 27;

  string web_url = 28;
  WebUrlInfo web_url_info = 29;

  // If set, overrides the runtime used by the function, either "runc" or "gvisor".
  string runtime = 30;

  string stub_name = 31;

  repeated VolumeMount volume_mounts = 33;

  uint32 allow_concurrent_inputs = 34;

  repeated CustomDomainInfo custom_domain_info = 35;

  string worker_id = 36; // For internal debugging use only.

  bool runtime_debug = 37; // For internal debugging use only.

  // TODO: combine into enum?
  bool is_builder_function = 32;
  bool is_auto_snapshot = 38;
  bool is_method = 39;
  bool is_checkpointing_function = 40;

  // Checkpoint and restore

  bool checkpointing_enabled = 41;

  message CheckpointInfo {
    string checksum = 1;
    CheckpointStatus status = 2;
  }
  CheckpointInfo checkpoint = 42;
  repeated ObjectDependency object_dependencies = 43;
}

Let’s Come back up, Breathe and Dive Deep Again

Following up on the hello world, what if there are multiple modal @stub.functions under one stub?

import modal

stub = modal.Stub("hello")

@stub.function()
def hello():
    print("hello, world!")

@stub.function()
def hello2():
    print("hello, world second time!")

then modal run lib/hello_local_entry.py fails with

so we need to specify which one to run. We have two optionsvia CLI: modal run lib/hello_local_entry.py::helloOr specify it in code using @stub.local_entrypoint decorator and modal run lib/hello_local_entry.py (no need for ::hello)

@stub.function()
def hello():
    print("hello, world!")

@stub.function()
def hello2():
    print("hello, world second time!")

@stub.local_entrypoint()
def main():
    hello.local()  # semantically is `hello()`

Simply calling hello() doesn’t work (anymore) and Modal has its own calling semantics depending on the environment*hello.local() as above

which outputs

*Or via hello.remote()

@stub.local_entrypoint()
def main():
    # hello.local()
    hello.remote()

There is a subtle difference in their output. Did you notice it? there is an extra None after hello, world! in the .remote() case. Why?

@stub.function()
def my_func(a):
    return a ** 2

@stub.local_entrypoint()
def main():
    assert list(my_func.map([1, 2, 3, 4])) == [1, 4, 9, 16]

@stub.function()
def my_func(a, b):
    return a + b

@stub.local_entrypoint()
def main():
    assert list(my_func.starmap([(1, 2), (3, 4)])) == [3, 7]

import modal

stub = modal.Stub("my-shared-app")

@stub.function()
def square(x):
    return x * x

import modal

stub = modal.Stub("another-app")
stub.square = modal.Function.from_name("my-shared-app", "square") # <-- NOTE: this must be deployed otherwise `modal run` won't find it

@stub.function()
def cube(x):
    return x * stub.square.remote(x)

@stub.local_entrypoint()
def main():
    assert cube.remote(42) == 74088

import time
import modal
from modal.functions import FunctionCall

stub = modal.Stub("cube-spawn")
stub.square = modal.Function.from_name(
    "my-shared-app", "square"
)  # <-- NOTE: this must be deployed otherwise `modal run` won't find it

@stub.function()
def spawn_square(x):
    call = stub.square.spawn(x)
    return {"call_id": call.object_id}

@stub.function()
def poll(call_id):
    fcall = FunctionCall.from_id(call_id)
    try:
        # 5 seconds timeout to simulate a long running job
        ret = fcall.get(timeout=5)
    except TimeoutError:
        print("waiting for result")
        return

    return ret

@stub.local_entrypoint()
def cube():
    call = spawn_square.remote(42)
    call_id = call["call_id"]
    assert call_id is not None
    ret = poll.remote(call_id)
    assert ret * 42 == 74088

@web_app.post("/parse")
async def parse(request: fastapi.Request):
    parse_receipt = Function.lookup("example-doc-ocr-jobs", "parse_receipt")

    form = await request.form()
    receipt = await form["receipt"].read()  # type: ignore
    call = parse_receipt.spawn(receipt)
    return {"call_id": call.object_id}

@web_app.get("/result/{call_id}")
async def poll_results(call_id: str):
    from modal.functions import FunctionCall

    function_call = FunctionCall.from_id(call_id)
    try:
        result = function_call.get(timeout=0)
    except TimeoutError:
        return fastapi.responses.JSONResponse(content="", status_code=202)

    return result

const resp = await fetch("/parse", {
      method: "POST",
      body: formData,
    });

...

const _intervalID = setInterval(async () => {
      const resp = await fetch(`/result/${callId}`);
      if (resp.status === 200) {
        setResult(await resp.json());
      }
    }, 100);

    setIntervalId(_intervalID);

from modal import Stub, method

stub = Stub()

@stub.cls()
class Model:
    def __enter__(self):
        self.model = pickle.load(open("model.pickle"))

    @method()
    def predict(self, x):
        return self.model.predict(x)

@stub.local_entrypoint()
def main():
    Model().predict.remote(x) # <-- it's not like context manager `with ...` at all!

Build time Dependencies and Resources

image = Image.debian_slim().pip_install("pandas", "numpy")
@stub.function(image=image)
def my_function():
    import pandas as pd
    import numpy as np
    ...

image = (
    Image.debian_slim(python_version="3.10")
    .pip_install(
        "optimum[onnxruntime-gpu]==1.7.3",
        "transformers[onnx-gpu]==4.28.1",
        gpu="A10G",
    )
)

Image.debian_slim().apt_install("ffmpeg")

Image.debian_slim().apt_install("curl").run_commands(
        "curl -O https://raw.githubusercontent.com/opencv/opencv/master/data/haarcascades/haarcascade_frontalcatface.xml",
)

Image.from_registry("huanjason/scikit-learn", add_python=3.10)

Image.conda().conda_install("theano-pymc==1.1.2", "pymc3==3.11.2")

image = modal.Image.debian_slim(python_version="3.10").poetry_install_from_file("pyproject.toml", without=["dev"])

Image.from_dockerfile("Dockerfile")

def download_models():
    from sentence_transformers import SentenceTransformer

    SentenceTransformer(settings.EMBEDDING_MODEL_NAME, cache_folder=EMBEDDING_MODEL_DIR)

image = (
    modal.Image.debian_slim(python_version="3.10")
    .env({"ENV": ENV})
    .run_commands("python -m pip install --upgrade pip")
    .poetry_install_from_file("pyproject.toml", without=["dev"])
    .pip_install(
        "torch==2.1.0+cu118",
        find_links="https://download.pytorch.org/whl/torch_stable.html",
    )
    .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})
    .env({"TRANSFORMERS_CACHE": EMBEDDING_MODEL_DIR})
    .run_function(download_models)
)

secret = modal.Secret.from_dotenv(".env"))
@stub.function(
    image=image,
    secret=secret)
def example():
    ....

import pathlib
import modal

p = pathlib.Path("/root/foo/bar.txt")

stub = modal.Stub()
stub.volume = modal.Volume.new()

@stub.function(volumes={"/root/foo": stub.volume})
def f():
    p.write_text("hello")
    print(f"Created {p=}")
    stub.volume.commit()  # Persist changes
    print(f"Committed {p=}")

import modal

stub = modal.Stub()
volume = modal.NetworkFileSystem.new()

@stub.function(network_file_systems={"/root/foo": volume})
def f():
    pass

@stub.function(network_file_systems={"/root/goo": volume})
def g():
    pass

@stub.function(
    image=image,
    secret=secret,
    concurrency_limit=4,
    cpu=8,
    gpu="t4",
    memory=1024 * 8
    timeout=5 * 60,
    keep_warm=True,
    cloud="aws", # as of now, 'aws', 'gcp' and oracle cloud are supported
)
@modal.asgi_app(label="server")
def main() -> Callable:
    app = get_app()
    return app

secret = modal.Secret.from_dotenv(".env"))
image = (
    modal.Image.debian_slim(python_version="3.10")
    .env({"ENV": ENV})
    .run_commands("python -m pip install --upgrade pip")
    .poetry_install_from_file("pyproject.toml", without=["dev"])
)

@stub.function(
    image=image,
    secret=secret,
    cpu=8,
    gpu="any",
    memory=1024 * 8,
    cloud="aws",
)
def main():
    ...

from modal import Stub, web_endpoint

stub = Stub()

@stub.function()
@web_endpoint()
def f():
    return "Hello world!"

from fastapi import FastAPI, Request
from fastapi.responses import HTMLResponse

from modal import Image, Stub, asgi_app

web_app = FastAPI()
stub = Stub()
image = Image.debian_slim().pip_install("boto3")

@web_app.post("/foo")
async def foo(request: Request):
    body = await request.json()
    return body

@web_app.get("/bar")
async def bar(arg="world"):
    return HTMLResponse(f"<h1>Hello Fast {arg}!</h1>")

@stub.function(image=image)
@asgi_app()
def fastapi_app():
    return web_app

modal launch jupyter/vscode

Modal CLI has the option oflaunching jupyter or vscode from modal by also configuring CPU/GPU and memory as needed. This essentially makes manual SSHing and to EC2 instancesobsolete. Jupyter is a great tool for exploratory part of ML such astraining. The vscode is just another cherry on top. Here I have launched vscode with T4 GPU using

modal launch vscode --gpu T4

The CLI command is basically running the following which you can customize as well.

# Copyright Modal Labs 2023
# type: ignore
import os
import secrets
import socket
import subprocess
import threading
import time
import webbrowser
from typing import Any

from modal import Image, Queue, Stub, forward

args: Any = {'cpu': 8, 'memory': 32768, 'gpu': 'T4', 'timeout': 3600}

stub = Stub()
stub.image = Image.from_registry("codercom/code-server", add_python="3.11").dockerfile_commands("ENTRYPOINT []")
stub.q = Queue.new()

def wait_for_port(data):
    start_time = time.monotonic()
    while True:
        try:
            with socket.create_connection(("localhost", 8080), timeout=15.0):
                break
        except OSError as exc:
            time.sleep(0.01)
            if time.monotonic() - start_time >= 15.0:
                raise TimeoutError("Waited too long for port 8080 to accept connections") from exc
    stub.q.put(data)

@stub.function(cpu=args["cpu"], memory=args["memory"], gpu=args["gpu"], timeout=args["timeout"])
def run_jupyter():
    os.chdir("/home/coder")
    token = secrets.token_urlsafe(13)
    with forward(8080) as tunnel:
        url = tunnel.url
        threading.Thread(target=wait_for_port, args=((url, token),)).start()
        subprocess.run(
            ["/usr/bin/entrypoint.sh", "--bind-addr", "0.0.0.0:8080", "."],
            env={**os.environ, "SHELL": "/bin/bash", "PASSWORD": token},
        )
    stub.q.put("done")

@stub.local_entrypoint()
def main():
    stub.run_jupyter.spawn()
    url, token = stub.q.get()
    time.sleep(1)  # Give Jupyter a chance to start up
    print("\nVS Code on Modal, opening in browser...")
    print(f"   -> {url}")
    print(f"   -> password: {token}\n")
    webbrowser.open(url)
    assert stub.q.get() == "done"

Best Practices

Since Modal is new there are almost no written best practices available. Here are my suggestions after using Modal for almost a year and trying many things

myapp /
    __init__.py
    stub.py
    app /
        api.py
    services /
        service1.py
        service2.py

import modal
from myapp.settings import get_settings

settings = get_settings()
stub = modal.Stub(settings.STUB_NAME)
stub.queue = modal.Queue.new().persisted(label=settings.STUB_QUEUE_NAME)

from myapp.api.app import *
from myapp.services.service1 import *
from myapp.services.service2 import *

from pathlib import Path
from typing import List
import numpy as np
import tqdm
import modal

from myapp.settings import get_settings
from myapp.stub import stub

settings = get_settings()
secret = modal.Secret.from_dotenv( ".env"))

def download_models():
    ...

image = (
    modal.Image.debian_slim(python_version="3.10")
    ...
    .run_function(download_models)
)

@stub.cls(image=image, secret=secret)
class BatchEmbedding:
    def __enter__(self):
        from sentence_transformers import SentenceTransformer

        self.model = SentenceTransformer(
            settings.EMBEDDING_MODEL_NAME,
            cache_folder=EMBEDDING_MODEL_DIR,
        )
        return self

    @modal.method()
    def embed(self, texts) -> np.ndarray:
        vectors = self.model.encode(
            texts,
            show_progress_bar=True,
            batch_size=settings.EMBED_BATCH_SIZE,
        )
        return vector

class DocumentEmbeddingFn:
    def __call__(self, document: Document) -> List[np.ndarray]:
        if not len(document):
            return []

        batches: List[List[TextDoc]] = [
            document.chunks[i : i + settings.EMBED_BATCH_SIZE]
            for i in range(0, len(document), settings.EMBED_BATCH_SIZE)
        ]
        chunked_batches: List[List[str]] = [[chunk.text for chunk in batch] for batch in batches if len(batch)]
        vectors: List[np.ndarray] = []
        for batch in tqdm(list(BatchEmbedding().embed.map(chunked_batches))):
            vectors.extend(batch)

        return vectors

if settings.ENV == "test":
    import io
    from pathlib import Path
    from fastapi import UploadFile

    @stub.local_entrypoint()
    async def test_embedding():
        logger.info("Testing embedding service with sample PDF file")
        data = Path(__file__) / "tests/data.pdf"
        with open(str(data), "rb") as fin:
            file = UploadFile(
                file=io.BytesIO(fin.read()),
                filename="data.pdf",
            )
        doc = ..
        embeddings = DocumentEmbeddingFn()(doc)
        assert len(embeddings) == len(doc)

poetry_install_from_file("pyproject.toml", without=["dev"])

[tool.poetry.dependencies]
python = ">=3.10,<3.12"
modal = "^0.52.3366"

[tool.poetry.core.dependencies]
numpy = "^1.20.0"
pydantic = "^2.3.0"

[tool.poetry.db.dependencies]
pymongo = { version = "^4.5", extras = ["srv"] }

[tool.poetry.service1.dependencies]
...

[tool.poetry.service2.dependencies]
...

poetry_install_from_file("pyproject.toml", with_=["core", "db", "service1"])

Acknowledgement

I want to thank Modal engineers in particular Akshat Bubna (Modal’s co-founder), Jonathon Belotti and Luis Capelo for their immense help, patient and being very responsive to my queries throughout using Modal. From my experience, Modal’s customer service is fantastic despite it being run by small developer team.

Rediscovering Attention

Interestingly, the of core attention hasnothingto do with Deep Learning.Attention is in fact a communication mechanism in a graph / network. Let me explain how. Given a directed graph , where is the set of nodes and a set of edges, we can associatetwopieces of information to each node ;(key, query)where key is the “identity” of the node, query is what thenode is looking for and is interested in finding. (Note there is no value piece yet). Take the following directed graph. The red arrows from the view point of node . The key is the self-arrow and the rest of the arrows make up the query piece. We haven’t yet defined key and query.

We can parameterize key and query and represent them as (learnable) vectors (in PyTorch nn.Parameter). So now looking at other nodes key vectors, the node can find its interest / “attention” score by for example computing cosine similarity (cosSim) between its query and the key of every other neighbours (connected to) i.e. 

and gathering all the similarity scores in a vector

So far so good! now what to do with this similarity vector? we need to include another piece which is thevalue. This value is the internal representation of nodes so for let’s denote it by (note it’s parameterized to be learned). Normally, we would start with some value. For example in the NLP, node / token embedding and adjust that iteratively. But for now, we can assume the value vector is given so we update it by the discovered similarity scores for example by element-wise multiplication

Note it doesn’t have to be element-wise multiplication but that seems to be the simplest case here. Now, recall

so

isalmostequivalent to

where is a matrix formed by stacked all keys in rows. Note here, we have to be careful about full vector-matrix multiplication because we need to only multiply to the keys of its neighbours. Let’s simplify and put that details aside. So as you can see the final rediscovered attention formula from the view point of a single node is

or putting the “weights” down

and going for all nodes by stacking their vectors into matrices, we will get at

it’s not hard to see it resembles the original attention formula

dim = 64
w_q = nn.Parameter(torch.randn(dim))
W_k = nn.Parameter(torch.randn(dim, dim))
w_v = nn.Parameter(torch.randn(dim))
attention_scores = w_q @ W_k
attention_weights = F.softmax(attention_scores, dim=0)
output = attention_weights @ w_v

In fact, was replaced with (and the additional normalization). Also it turns out the version is calledContent-base attentionand is used in Neural Turing Machine. In fact, others have tried different “adjustments” to the formula as highlighted here.

How does this formulation map to graphs from NLP, CV etc?

NLP and Vision make explicit use of it by creating the graph suitable for their tasks. For example, in (Causal) Language Model (LM) the core of (GPT) attention graph looks like

It is auto-regressive i.e. only past informations / edges are allowed so with a proper mask (lower-triangular matrix) we can zero out the future information

attention_scores = w_q @ W_k
tril = torch.tril(torch.ones(T, T)) # lower-triangle matrix of 1s
attention_scores = attention_scores.masked_fill(tril == 0, float('-inf')) # masks the future and set to -inf
attention_weights = F.softmax(attention_scores, dim=0)
output = attention_weights @ w_v

Or in Vision, the attention graph of the vision transformer model (ViT) from An Image is Worth 16 x 16 Words looks like this (complete graph i.e. all nodes are connected to each other)

Summary

We tried to rediscover the attention formula and did it by putting the attention in a higher context i.e.communication in a graph. Note that depending on the task and the graph at hand we need to make adjustments like in the LM case, by masking out future information. Also the general notion of Attention has no notion of position and that is why encoding and incorporating positions i.e. positional encoding in LM is important. The complexity grows from here and we can create Multi-Head Attention and even more general Cross-Attention (with encoder-decoder). But these are less important than the core intuition I wanted to give in this post. Hope this post has clarified attention and the intuition behind it. If you are interested in more detail implementation of the Attention-is-all-You-Need paper, I recommend checking out annotated implementation in labml.

Cheat Table

table { border-collapse: collapse; width: 100%; } th, td { border: 1px solid black; padding: 8px; text-align: left; }

Resize image example

For another example, let’s say you want to enhance resizing jpeg image.

1. npm init neon image-resize and cd image-resize
2. Then cargo add image and
3. Include the following in src/lib.rs. Check out the comments below

use std::io::Cursor;
use image;
use neon::{prelude::*, types::buffer::TypedArray};
fn resize_image(mut cx: FunctionContext) -> JsResult<JsBuffer> {
    // Retrieve image buffer and dimensions from JavaScript arguments
    let buffer = cx.argument::<JsBuffer>(0)?; // <- gets the first argument in node
    let width = cx.argument::<JsNumber>(1)?.value(&mut cx) as u32;
    let height = cx.argument::<JsNumber>(2)?.value(&mut cx) as u32;
    // Convert JS Buffer to a byte slice
    let image_data: &[u8] = buffer.as_slice(&cx);
    // Perform image resizing in rust
    let img = image::load_from_memory(&image_data).expect("Failed to load image from memory");
    let resized = img.resize(width, height, image::imageops::FilterType::Nearest);
    let mut resized_buffer = Cursor::new(Vec::new());
    resized
        .write_to(&mut resized_buffer, image::ImageOutputFormat::Jpeg(100))
        .expect("Failed to write image to buffer");
    let img_data = resized_buffer.into_inner();
    // Convert the byte vector back to a JS Buffer
    let js_buffer = JsBuffer::external(&mut cx, img_data);
    Ok(js_buffer)
}
// finally export the function as a module
register_module!(mut cx, { cx.export_function("resizeImage", resize_image) });

4. npm install

5. Run node prompt and test with an image

const nativeModule = require('.');
const fs = require('fs');
let imageBuf = fs.readFileSync('cat.jpeg');
const resizedBuf = nativeModule.resizeImage(imageBuf, 50, 50);
fs.writeFileSync('resized_cat.jpeg', resizedBuf);

neon also provides a lot of “promise-api”, “task-api” to handle js promise and node worker pool jobs.

Segment-Anything in Node

Our examples won’t be complete with some deep learning heavy computation. We use Segment-Anything which is a state-of-the-art segmentation model by Meta. It produces high quality object masks from input prompts such as points or boxes, and it can be used to generate masks for all objects in an image. We will using Huggingface Candle Rust Deep Learning library as the final example that delegates heavy lifting to Rust.

1. npm init neon sam-node and cd sam-node
2. Include the dependencies in Cargo.toml as follows

[dependencies]
anyhow = "1.0.75"
candle-core = { git = "https://github.com/huggingface/candle.git", version = "0.3.1", feautures = [
    "accelerate",
] }
candle-examples = { git = "https://github.com/huggingface/candle.git", version = "0.3.1", feautures = [
    "accelerate",
] }
candle-nn = { git = "https://github.com/huggingface/candle.git", version = "0.3.1", feautures = [
    "accelerate",
] }
candle-transformers = { git = "https://github.com/huggingface/candle.git", version = "0.3.1", feautures = [
    "accelerate",
] }
hf-hub = "0.3.2"
image = "0.24.7"
imageproc = "0.23.0"

and make sure you may need to set Huggingface API key if you haven’t.

3. Now in src/lib.rs

fn generate_sam(mut cx: FunctionContext) -> JsResult<JsString> {
    let image_path = cx.argument::<JsString>(0)?.value(&mut cx);
    let points = cx.argument::<JsArray>(1)?;
    let neg_points = cx.argument::<JsArray>(2)?;
    let points = get_points(&mut cx, points);
    let neg_points = get_points(&mut cx, neg_points);
    _generate_sam(image_path, points, neg_points).expect("error generating sam");
    Ok(cx.string("ok"))
}
fn get_points(cx: &mut FunctionContext, handle: Handle<JsArray>) -> Vec<String> {
    let points = handle.to_vec(cx).expect("error converting to vec");
    let mut ret: Vec<String> = Vec::new();
    for point in points {
        let point_string = point
            .downcast::<JsString, FunctionContext>(cx)
            .or_else(|_| cx.throw_error("Array element is not a string"))
            .unwrap()
            .value(cx);
        ret.push(point_string);
    }
    // return as a vec but a single contiguous string
    ret = vec![ret.join(",")];
    ret
}

where

fn _generate_sam(
    image_path: String,
    points: Vec<String>,
    neg_points: Vec<String>,
) -> anyhow::Result<()> {
    let device = candle_examples::device(true)?; // use CPU
    let (image, initial_h, initial_w) =
        candle_examples::load_image(&image_path, Some(sam::IMAGE_SIZE))?;
    let image = image.to_device(&device)?;
    println!("loaded image {image:?}");
    let api = hf_hub::api::sync::Api::new()?;
    let api = api.model("lmz/candle-sam".to_string());
    let filename = "mobile_sam-tiny-vitt.safetensors";
    let model = api.get(filename)?;
    let vb = unsafe { VarBuilder::from_mmaped_safetensors(&[model], DType::F32, &device)? };
    let sam = sam::Sam::new_tiny(vb)?;
    // Default options similar to the Python version.
    let bboxes = sam.generate_masks(
        &image,
        /* points_per_side */ 32,
        /* crop_n_layer */ 0,
        /* crop_overlap_ratio */ 512. / 1500.,
        /* crop_n_points_downscale_factor */ 1,
    )?;
    for (idx, bbox) in bboxes.iter().enumerate() {
        println!("{idx} {bbox:?}");
        let mask = (&bbox.data.to_dtype(DType::U8)? * 255.)?;
        let (h, w) = mask.dims2()?;
        let mask = mask.broadcast_as((3, h, w))?;
        candle_examples::save_image_resize(
            &mask,
            format!("sam_mask{idx}.png"),
            initial_h,
            initial_w,
        )?;
    }
    let iter_points = points.iter().map(|p| (p, true));
    let iter_neg_points = neg_points.iter().map(|p| (p, false));
    let points = iter_points
        .chain(iter_neg_points)
        .map(|(point, b)| {
            use std::str::FromStr;
            let xy = point.split(',').collect::<Vec<_>>();
            if xy.len() != 2 {
                anyhow::bail!("expected format for points is 0.4,0.2")
            }
            Ok((f64::from_str(xy[0])?, f64::from_str(xy[1])?, b))
        })
        .collect::<anyhow::Result<Vec<_>>>()?;
    let start_time = std::time::Instant::now();
    let (mask, iou_predictions) = sam.forward(&image, &points, false)?;
    println!(
        "mask generated in {:.2}s",
        start_time.elapsed().as_secs_f32()
    );
    println!("mask:\n{mask}");
    println!("iou_predictions: {iou_predictions}");
    let mask = (mask.ge(0.)? * 255.)?;
    let (_one, h, w) = mask.dims3()?;
    let mask = mask.expand((3, h, w))?;
    let mut img = image::io::Reader::open(&image_path)?
        .decode()
        .map_err(candle::Error::wrap)?;
    let mask_pixels = mask.permute((1, 2, 0))?.flatten_all()?.to_vec1::<u8>()?;
    let mask_img: image::ImageBuffer<image::Rgb<u8>, Vec<u8>> =
        match image::ImageBuffer::from_raw(w as u32, h as u32, mask_pixels) {
            Some(image) => image,
            None => anyhow::bail!("error saving merged image"),
        };
    let mask_img = image::DynamicImage::from(mask_img).resize_to_fill(
        img.width(),
        img.height(),
        image::imageops::FilterType::CatmullRom,
    );
    for x in 0..img.width() {
        for y in 0..img.height() {
            let mask_p = imageproc::drawing::Canvas::get_pixel(&mask_img, x, y);
            if mask_p.0[0] > 100 {
                let mut img_p = imageproc::drawing::Canvas::get_pixel(&img, x, y);
                img_p.0[2] = 255 - (255 - img_p.0[2]) / 2;
                img_p.0[1] /= 2;
                img_p.0[0] /= 2;
                imageproc::drawing::Canvas::draw_pixel(&mut img, x, y, img_p)
            }
        }
    }
    for (x, y, b) in points {
        let x = (x * img.width() as f64) as i32;
        let y = (y * img.height() as f64) as i32;
        let color = if b {
            image::Rgba([255, 0, 0, 200])
        } else {
            image::Rgba([0, 255, 0, 200])
        };
        imageproc::drawing::draw_filled_circle_mut(&mut img, (x, y), 3, color);
    }
    img.save("sam_merged.jpg")?;
    Ok(())
}
// finally register as node module
register_module!(mut cx, { cx.export_function("generateSam", generate_sam) });

Next, after sam-node/index.js we add simple Node.js express server. (Note the dependencies npm install axios express)

const express = require('express');
const nativeModule = require('.');
const app = express();
const port = 3000;
const axios = require('axios');
const fs = require('fs');
const path = require('path');
async function downloadImage(url, filePath) {
  try {
    const response = await axios({
      method: 'GET',
      url: url,
      responseType: 'stream'
    });
    const writer = fs.createWriteStream(filePath);
    response.data.pipe(writer);
    return new Promise((resolve, reject) => {
      writer.on('finish', resolve);
      writer.on('error', reject);
    });
  } catch (error) {
    console.error('Error downloading the image:', error);
    throw error;
  }
}
app.use(express.json());
app.listen(port, () => console.log(`Listening on port ${port}`));
app.post("/generate-sam", async (req, res) => {
    try {
        const { imagUrl, points, negPoints } = req.body;
        const name = imagUrl.split('/').pop();
        const filePath = path.join(__dirname, name);
        await downloadImage(imagUrl, filePath);
        await nativeModule.generateSam(filePath, points, negPoints);
    } catch (error) {
        console.log(error);
        res.status(500).send(error);
    }
});

Then node index.js and can test with npm install node-fetch script in test.js

that downloads the sample JPG image

// testing using node-fetch to send a POST request to the server
const url = 'http://localhost:3000/generate-sam';
const imageUrl = 'https://githubraw.com/huggingface/candle/main/candle-examples/examples/yolo-v8/assets/bike.jpg';
const points = ['0.6', '0.6'];
const negPoints = ['0.6', '0.55'];
fetch(url, {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json',
    },
    body: JSON.stringify({
        imagUrl: imageUrl,
        points: points,
        negPoints: negPoints
    })
})
.then(response => {
    if (!response.ok) {
        throw new Error('Network response was not ok');
    }
    return response.blob();
})
.then(blob => {
    console.log('Image received:', blob);
    const imageUrl = URL.createObjectURL(blob);
    const img = document.createElement('img');
    img.src = imageUrl;
    document.body.appendChild(img);
})
.catch(error => {
    console.error('Fetch error:', error);
});

where segmentation was applied to the right-most cyclist’s right foot.

Hope this post has ignited some spark to explore neon and the Rust-Node.js interactions.

Library vs. Framework

It’s tricky to draw a clear boundary between a package/library and a framework, but for the sake of discussion, let’s look at some well-known examplesPackages:Numpy falls into this category. It provides functionality that can be adapted to various Linear Algebra problems without dictating a particular structure or methodology.Mature Frameworks: Scikit-learn, PyTorch and HuggingFace transformers. They provide high-level abstractions but take customization and non-opinionated design seriously. As a result, they have become the de facto ways of doing ML/AI.ImmatureFrameworks:**LangChain (LC) and Llama-index (LI) are examples in this category. Unlike packages, they offer higher-level abstractions and impose a specific way of building software. (Within this category, I’ve found that LI provides better abstractions than LC).

I’ve been using LC and LI for seven months now, and their evolution exemplifies a broader trend in the ML/AI world. While they’ve been great for quickly creating a Proof-of-Concept, their higher-level abstractions miss many details and nuances of components that are hard to include in a clean abstraction.

Take VectorDB/VectorStore; within a few months, their numbers have grown quickly. The quality of each is up to the individual to decide and try (I have my own suggestions, perhaps for later). For a long time, none of these frameworks offered a simple, clear CRUD API. Also, each VectorDB has its nuances of handling embeddings, from storage to search. Some support async, and some don’t. Some support gRPC, and some don’t. Some support hybrid-search, and some don’t. To use them at scale for production, we need to know about these nuances to squeeze every bit of performance. So what LC and LI did was provide a base VectorStore class and implement methods like add_documents (and their async aadd_documents) and wrap them without exposing the individual nuances of VectorDBs.

Mostly in LC, almost every abstracted away component is opinionated and hard to customize. I can add async, streaming, callback-hell, agent and memory to the list. Not everything needs to be a subclass of pydantic BaseModel! I think “true chaining” is not there yet in LC (should admit overloading __or__ is an interesting new way, similar to Unix pipe), and the many ways of calling a chain/agent with run, __call__, apply (now with added async) are unhealthy. These criticisms of LC are mostly to the point but I also think given the rapid change of landscape both LC and LI provide values esp. as a generic off-the-shelf solutions.

Conclusion (Updated)

I don’t think these frameworks, esp. LC offers anything good long term. LI has a better chance of remaining relevant but after spending some time, I came into conclusion that many of the framework decisions were taken that are hard to change and won’t work with down the road esp. if you application needs much higher control over how things are done such as preserving the structure of the documents during chunking with non-flat node/graph system, granular control over embedding of different data types, handling VectorDBs components, let alone the “data” agent etc.

Pin and Unpin

#[fundamental] // compiler thing
#[repr(transparent)] // --> ensures same memory repr as P
pub struct Pin<P> {
    pointer: P,
}
pub auto trait Unpin { }  // marker trait

To construct a Pin safely via Pin::new(pointer: P), we need to make sure that P: Deref (pointer-like) and P::Target: Unpin.

Note that Unpin ensures

Types which can besafely moved after being pinned.

Since Rust itself has no notion of immovable types, and considers moves (e.g. through assignment or mem::replace) to always be safe, this trait cannot prevent types from moving by itself.
Instead it is used toprevent moves through the type system, by controlling the behavior of pointers P wrapped in the Pin<P> wrapper, which “pin” the type in place by not allowing it to be moved out of them.

Rust std doc

Basically, the vast majority of types are “movable” i.e. Unpin, so for most cases,Pin<Pointer<T>>is exactly likePointer<T>whenT: Unpin. However, if the referent (referenced type T) is “immovable” i.e. T: !Unpin, then the pinned wrapper won’t let the underlying value to move.*Intuitively it’s like having**&mut* but with & access safety and immovability invariant. It is in fact, Unpin that controls how Pin to work.

To iterate; Pin<P> ensures that theunderlying data behind a pointer typePhas a stable location in memoryandits memory cannot be deallocated until it is dropped. So Pin pins the pointee (the data behind the pointer)not the pointer itself. Moreover,Pin<P>does not let clients actually obtain aBox<T>or&mut Tto the underlying pinned data, which implies that youcannotuse operations such as mem::swap.

Furthermore,

For correctness, Pin<P> relies on the Deref and DerefMut implementations tonot move out of theirselfparameter, and to only ever return a pointer to pinned data when they are called on a pinned pointer.

Rust std doc

To implement !Unpin for a type directly, (up to this write up) we need nightly with ![feature(optin_builtin_traits)]. The stable way is through the marker type PhantomPinned which already has impl !Unpin for PhantomPinned {}

Closer look

We cannot create Pin for a Pointer<T> where T: !Unpin safely

pub struct Pin<P> {
    pointer: P,
}
impl<P: Deref> Pin<P> 
where P::Target: Unpin // --> underlying data is movable
{
    pub fn new(pointer: P) -> Pin<P> {
        // Safety: the value pointed to is `Unpin`, and so has no requirements
        // around pinning.
        unsafe { Pin::new_unchecked(pointer) }
    }
}
impl<P: Deref> Pin<P> {
   pub unsafe fn new_unchecked(pointer: P) -> Pin<P> {
        Pin { pointer }
    }
}

impl<T> Box<T> {
    #[inline(always)]
    pub fn pin(x: T) -> Pin<Box<T>> {
        (box x).into()
    }
}
impl<T: ?Sized> From<Box<T>> for Pin<Box<T>> {
    /// This conversion does not allocate on the heap and happens in place.
    fn from(boxed: Box<T>) -> Self {
        Box::into_pin(boxed)
    }
}
impl<T: ?Sized> Box<T> {
    pub fn into_pin(boxed: Box<T>) -> Pin<Box<T>> {
        // It's not possible to move or replace the insides of a `Pin<Box<T>>`
        // when `T: !Unpin`,  so it's safe to pin it directly without any
        // additional requirements.
        unsafe { Pin::new_unchecked(boxed) }
    }
}

impl<P: DerefMut> Pin<P> {
    pub fn set(self: &mut Pin<P>, value: P::Target)
    where
        P::Target: Sized,
    {
        *(self.pointer) = value; // --> before assignment the pointee *self.pointer is dropped
    }
}