shykes

Dagger Parallel Execution - Full Design Conversation

User

Help me plan a new "collaborative parallel execution" feature in dagger. The general idea:

• Dagger functions can be annotated with a new directive @parallel (+parallel in Go etc) which indicates to the engine that this function can be executed in parallel
• The dimensions of paralellism are controlled by arguments. Any argument to a parallel function that is a list, is interpreted by the engine as a dimension of parallelism. The contract is that when a client invokes a paralle function, the engine may split that "frontend" invocation into N "backend" invocations, with each invocation receiving a slice of the frontend list argument. The function implementor is responsible for guarranteeing correct behavior of the function regardless of the splitting.
• Only return types that can be reliably merged in any order (I don't know the fancy terminology) is allowed. Example: Directory is allowed because you can always merge N directories together. Chan

shykes

Collaborative Parallel Execution in Dagger

Summary

This document explores design options for a new parallel execution feature in Dagger, enabling functions to be automatically split and executed across multiple workers.

---

Problem Statement

shykes

registryURL: string

runRegistry: docker.#Run & {
        args: ["registry", "-h", registryURL] // bla bla
        // Wait for registry port to be available
        hooks: post: waitRegistry: {
                args: """
                        while true; do
                                curl -L -S "localhost:foo" && exit
                                sleep 1

shykes

#!/bin/bash

set -ex

update() {
	mkdir -p ~/bin
	curl -L -o ~/bin/dev.sh https://gist.githubusercontent.com/shykes/7bf0704be14a0a9288d7dfe98617c208/raw/dev.sh
	chmod +x ~/bin/dev.sh
}

shykes

Keybase proof

I hereby claim:

• I am shykes on github.
• I am shykes (https://keybase.io/shykes) on keybase.
• I have a public key ASAG6dMVHHD2CVy-aMBp-QRCYo--xvC2O4aQXhHUTN7dYQo

To claim this, I am signing this object:

shykes

FROM ubuntu
RUN apt-get update -y
RUN apt-get install -y gcc
RUN gcc -o /src/myapp /src/myapp.c
IN /target {
  FROM busybox
  ENTRYPOINT /myapp
}
RUN cp /src/myapp /target/myapp
PUBLISH /target

shykes

Simple ambassador container

The Simple Ambassador Container (let's just call it the ambassador) is a reusable container which can be added to your stack to represent a remote service. Using an ambassador your application can discover remote services using docker's standard links features.

Usage

Example usage:

REDIS_REMOTE_IP=1.2.3.4

shykes

type Driver interface {
        Create(id, parent string) error
        Remove(id string) error

        Get(id string) (dir string, err error)

        Diff(id string) (archive.Archive, error)
        DiffSize(id string) (bytes int64, err error)
        Changes(id string) ([]Change, error)

shykes

interface GraphBackend {
  Init(home string) error
  Nuke() error
  Create(img *Image, layer Archive) error
  Delete(img *Image) error
  Mount (img *Image, dest string) error
  UnmountAll (img *Image) error
  DiffList(*Image, dest string) ([]Change, error)
  DiffLayer(*Image, dest string) (Archive, error)
}

shykes

Links - Spec

Links are a way of expressing a relationship between one container and another. When you link a container, you create a parent/child relationship between the two. As a side effect links also allow you to name containers and reference them by name. Containers can be references by more than one name.

Initially links will be expressed by injecting the child's data into the parent's environment accessible via namespaced variables:

####Examples

• REDIS_ENV_{key}={value}
• REDIS_PORT={value}