Injecting environment variables

With the --env or -e flags.
A nice trick to see all the environment variables in a docker container is to use the Unix command env

Setting multiple environment variables: use \ for multiline like this:

docker create \
  --env WORDPRESS_DB_HOST=<my database hostname> \
  --env WORDPRESS_DB_USER=site_admin \
  --env WORDPRESS_DB_PASSWORD=MeowMix42 \ 
wordpress:4

Automatically restarting containers

Docker uses an exponential backoff strategy - double the previous time waiting until restarting.

docker run -d --restart always ...

See these restart policies

• no
• on-failure[:max-retries]
• always
• unless-stopped

Removing containers vs. images

Containers are the actual instantiation of an image, just like how an object is an instantion of an instance of a class.

docker rm: remove a container docker rmi: remove an image

Images as files

You can transport, save and load images as files! (You don’t have to push them to a registry).

You can then load the image:

docker load -i myfile.tar

Different kind of Volumes

1. Bind mount - this is what you always use
2. Docker managed volume (2 kinds)
   1. Anonymous
   2. Named volume (a special case of Anonymous)

Use volumes | Docker Documentation - Named vs. Anonymous volumes: article - Hamel: maybe? You might use named volumes to persist data between containers.

To persist data with named volumes

Named volume is a kind of anonymous volume where the mount point is managed by Docker. Example of how you used a named volume:

1. Start container with a named volume: docker run --name myDatabaseWithVolume -v appdata:/var/lib/mysql mysql save a table in the mysql database

2. Start a new container with the same named volume docker run --name myDatabaseWithVolume2 -v appdata:/var/lib/mysql mysql You should be able to see the same table you created in the last container b/c data has been persisted.

See where Docker anonymous volumes store information

Unlike a bind mount, where you explicitly name the host location, docker will manage the storage location of anonymous volumes. But how do you know where the files are stored on the host?

You can use docker inspect command filtered for the Volumes key to find the storage location on the host.

Create a container with an anonymous volume. docker run -v /some/location --name cass-shared alpine

docker inspect -f "{{json .Volumes}}" cass-shared

This will output a json blob which will show the mount points.

Other things you didn’t know about volumes

• when you mount a volume, it overrides any files already at that location
  • You can mount specific files which avoid this
  • if you specify a host directory that doesn’t exist Docker will create it for you
    • exception: If you are mounting a file instead of a directory and it doesn’t exist on the host, Docker will throw an error
• you can mount a volume as read only -v /source:/destination:ro
  • see docs (there is this optional third argument for volumes)

The volumes-from flag

Allows you to share volumes with another container. When you use this flag, the same volumes are mounted into your container at the same location.

Volumes are copied transitively, so this will automatically mount volumes that are also mounted this way from another container.

Caveats - You cannot mount a shared volume to a different location within a container. This is a limitation of --volumes-from - If you have a collision in the destination mount point among several volumes-from only one volume will survive, which you can ascertain from docker inpsect - see above for how to use docker inspect - You cannot change the write permission of the volume, you inherit whatever the permission is in the source container.

Cleaning up volumes

-v flag

docker rm -v will delete any managed volumes referenced by the target container

However, if you delete all containers but forget a -v flag you will be left with an orphaned volume. This is bad b/c it takes up disk space until cleaned up. You have to run complicated cleanup steps to get rid of orphans.

Solution: There is none, its a good habit to use -v anytime you call docker rm

Hamel: this means that- - Don’t use managed volumes unless you really need it - If you do use them, try to include makefiles that include -v as a part of things

Advanced Volume Stuff

• You can have a volume container p. 72 so that you can reference --volume-from from all your containers.
  
• Data-paced volume containers, you can pre-load volume containers with data p. 73
• You can change the behavior of currently running containers by mounting configuration files and application in volumes. In a way, Hamel

exposing ports

-p 8080 This binds port 8080 to a random port on the host! you can find the port that the container is bound to by docker port <image name> example: docker run -p 8080 --name listener alpine docker port listener

This will give you output that looks like container --> host (which is reverse the other nomenclature of host:container

-p 8080:8080 this binds the container’s port to the host’s port 8080

-p 0.0.0.0:8080:8080/tcp same as above but specifies the interface and the tcp protocol.

Syntax is -p <host-interface>:<host-port>:<target-port>/<protocol>

Limit resources: Memory, CPU,

• -m or --memory
  • number, where unit = b, k, m or g
  • memory limits are not reservations, just caps
• --cpu-shares
  • is a weight you set that is used to calculate % of CPU usage allowed
  • % is calculated as weight / (sum of all weights)
  • only enforced when there is contention for a CPU
• --cpuset-cpus : limits process to a specific CPU
  • docker run -d --cpuset-cpus 0 Restricts to CPU number 0
  • Can specify a list or 0,1,2 or a range 0-2
• --device
  • mount your webcam: docker run --device /dev/video0:/dev/video0
• Shared memory : Hamel this was too advanced for me

Running as a user

• You can only inspect the default run-as User by creating or pulling the image
  • see p. 113
• Change run-as user
  • docker run --user nobody
  • The user has to exist in the image when doing this or you will get an error. The user will not be created automatically for you.
  • See available users:
    • docker run --rm busybox:latest awk -F: '$0=$1' /etc/passwd

Privileged Containers: TRY NOT TO DO THIS

• This is how you run Docker-in-Docker
• Priviliged containers have root privileges on the host.
  
• --privilged on docker create or docker run

Recovering changes to a stopped container

I always thought you have to commit changes in order to preserve changes to an image you made in a container. This is not true (although committing changes is a good idea).

Any changes you make to a container is saved even if the container is exited

To recover changes to a container

1. Find the container (if you didn’t name it with docker run --name it will be named for you), using docker ps -a
2. Start the container using docker start -ai <container_name> the -ai flags mean to attach and run interactively
3. Now you are in the container you can verify that everything you installed is still there!

Note: if you run your container initially with docker run --rm this automatically removes your container upon exit, so this might not be recommended as your changes are not recoverable if you forget to commit

Seeing changes to a container from the base image

docker diff <container name> will output a long list of of file changes: - A: file added - D: file deleted - C: file changed

Other tricks

You can override the entry point to the container permanently by using the --entrypoint flag: docker run --entrypoint

Understanding Images & Layers

• files are stored in a Union file system, so they are stored in specific layers. The file system you are seeing as an end user are a union of all the layers. Each time a change is made to a union file system, that change is recorded on a new layer on top of all of the others. The “union” of all of those layers, or top-down view, is what the container (and user) sees when accessing the file system.
  • This means if you are not careful you can bloat the file system by making a bunch of unnecessary changes to add/delete files.
• docker commit commits the top-layer changes to an image, meaning all the files changes are saved.

See image size with

docker images. Even though you remove a file, the image size will increase! This is because of the Union File System

See size of all layers

docker history <image name>

flatten an image This is kind of complicated, you can do this by exporting and importing the filesystem into a base image See pg. 140. BUT there is an experimental feature called docker build --squash -t <image> .You can enable experimental features by following these instructions: dockerd Docker Documentation. For Mac, you can turn on experimental features by setting experimental: true in `settings> Command Line > enable experimental

ENTRYPOINT something arugment vs. ENTRYPOINT [“something”, “argument”]

TLDR; use the ugly list approach

There are two instruction forms shell form and exec form docker - Dockerfile CMD shell versus exec form - Stack Overflow

The ENTRYPOINT instruction has two forms: the shell form and an exec form. The shell form looks like a shell command with whitespace-delimited arguments. The exec form is a string array where the first value is the command to execute and the remaining values are arguments. .

Most importantly, if the shell form is used for ENTRYPOINT, then all other arguments provided by the CMD instruction or at runtime as extra arguments to docker run will be ignored. This makes the shell form of ENTRYPOINT less flexible.

Other commands can use the exec form too! You must use the exec form when any of the arguments contain a whitespace:

FROM dockerinaction/mailer-base:0.6 
COPY ["./log-impl", "${APPROOT}"] 
RUN chmod a+x ${APPROOT}/${APP} && \ chown example:example /var/log 
USER example:example 
VOLUME ["/var/log"]  # each value in this array will be created as a new volume definition
CMD ["/var/log/mailer.log"]

Note: you usually don’t want to specify a volume at build time.

CMD vs. ENTRYPOINT (You should really try to always use both!)

CMD is actually an argument list for the ENTRYPOINT.

• Logically when you run a container it runs as <default shell program> ENTRYPOINT CMD
• You can override the ENTRYPOINT with docker run --entrypoint, and you can override commands by just passing commands to docker run : docker run <image name> <command>

FROM ubuntu

ENTRYPOINT [ "ls" ]
CMD ["-lah"]

As you can see using ENTRYPOINT as well as CMD separately provides your downstream users with the most flexibility.

COPY vs ADD

Use COPY. ADD has additional functionality like ability to download from urls and decompress files, which proved opaque over time and you shouldn’t use it.

ONBUILD

The ONBUILD instruction defines instructions to execute if the resulting image is used as a base for another build. those ONBUILD instructions are executed after the FROM instruction and before the next instruction in a Dockerfile.

FROM busybox:latest 
WORKDIR /app RUN touch /app/base-evidence 
ONBUILD RUN ls -al /app

Other Stuff

• You should always validate the presence of required environment variables in a startup shell script like entrypoint.sh

Docker Digests

Reference the exact SHA of a Container which is the only way to guarantee the image you are referencing has not changed. @ symbol followed by the digest.

Hamel: doesn’t look like a good way to find history of digests, but you can see the current SHA when you use docker pull , you can see the SHA as well if you call docker images --digests

FROM debian@sha256:d5e87cfcb730...

Scaling Up w/Docker Compose

That’s right you don’t need docker swarm. This example uses ch11_coffee_api/docker-compose.yml at master · dockerinaction/ch11_coffee_api · GitHub

1. Get list of containers that are currently providing the service.

docker-compose ps coffee

          Name                 Command       State            Ports
----------------------------------------------------------------------------
ch11_coffee_api_coffee_1   ./entrypoint.sh   Up      0.0.0.0:32768->3000/tcp

2. Scale it up with docker-compose up --scale

docker-compose up --scale coffee=5

When you run docker-compose ps coffee:

docker-compose ps coffee                                                                                                                         ✔
          Name                 Command       State            Ports
----------------------------------------------------------------------------
ch11_coffee_api_coffee_1   ./entrypoint.sh   Up      0.0.0.0:32768->3000/tcp
ch11_coffee_api_coffee_2   ./entrypoint.sh   Up      0.0.0.0:32769->3000/tcp
ch11_coffee_api_coffee_3   ./entrypoint.sh   Up      0.0.0.0:32771->3000/tcp
ch11_coffee_api_coffee_4   ./entrypoint.sh   Up      0.0.0.0:32770->3000/tcp
ch11_coffee_api_coffee_5   ./entrypoint.sh   Up      0.0.0.0:32772->3000/tcp

Note that the coffee service binds to port 0 on your host, which is an ephemeral port, which just means that your host machine assigns the service to a random port. This is required if you plan on using docker compose up --scale

The service was bound to port 0 on the host with

coffee:
  build: ./coffee
  user: 777:777
  restart: always
  expose:
    - 3000
  ports:
    - "0:3000"
...

3. Load balancer

Problem with this kind of scaling is you don’t know the ports in advance , and you don’t want to hit these individual endpoints, you need a load balancer. This blog post shows you how to luse NGINX as a load balancer.

You will need something like this in your compose file

  nginx:
    image: nginx:latest
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - pspdfkit
    ports:
      - "4000:4000"

Templating Docker Compose Files

You can read about this here: Share Compose configurations between files and projects | Docker Documentation, allows you to override certain things from a base compose file.

Focused vs Diffused Mode

You can not access focus and diffused mode simultaneously.

People have tried to access diffuse mode of thinking by bringing themselves to the point of sleep and waking up just as they fall asleep. For example, Salvador Dali - holding keys in your hand, and let the sound of keys falling the ground wake you up.

Exercise, going for a walk good way to access diffuse thinking. You must take notes right away b/c diffuse thoughts may evaporate very fast.

Procrastination Memory and Sleep

They advocate the Pomodoro technique to combating procrastination. Its like HITT.

Periodic relaxation (every ~ 30 minutes) is important for accessing your diffuse mode. “Its important for the mortar to dry”.

Spaced repetition (like Anki) is important for building memory. i

Go over what you want to learn about right before you go to sleep, this will substantially improve the chances you will dream about it and form new connections about the subject.

Exercise can help create new neurons in your hippocampus (new neurons can be created there in adulthood) and help them survive longer.

Writing Tips Diffuse Mode

Diffuse mode is very important for writing. Editing is like focus mode and creating ideas is diffuse mode. Some rules of thumb: - Do not outline, make a mind map - Do not edit while you are writing (this is really hard to do -> turn off monitor and just write).https://writeordie.com - app that forces you to stay in diffuse mode. You really cannot look at the screen. - Repeating again, do not look at screen while you are writing! Only when editing.

Chunking

“Tying your shoes”. Best chunks are subconscious. Spoken language is the best example of chunking. You have to practice to build chunks, you cannot just observe. You have to perform the task yourself.

You should scan a chapter before you read it: section headings, pictures, etc. This can help you build chunks.

Illusions of competence

Right after you read something, look away and repeat to yourself what you recall. You can also draw a concept map. The recall process actually improves memory.

Recall is better than re-reading. Re-reading is effective when you let time pass so you get spaced repetition. You need to test yourself to make sure you are competent. Recall is a form of testing.

Recall outside your place of study to strengthen your memory. This is because you can get queues from where you are studying.

Deliberate Practice

Focus on the bits that you find difficult. Interleaving is important, meaning learning different subjects or even sections within one subject at once. Thomas S. Khun discovered that two types of people tend to make scientific breakthroughs: (1) young people (2) those who are trained in another discipline.

Procrastination and Memory

You have already learned about the Pomodoro technique. There are other techniques.

Focus on the process, not the product. Don’t focus on completing the homework, focus on the process that leads you to complete the homework. Process is the small chunks of time to chip away at the task. This is the idea behind the Pomodoro. Your only goal is to finish the Pomodoro, for example.

Juggling Life and Learning

You should make to-do list the night before for the next day and write it down. This will allow your subconscious to work on how it will conquer that task. Furthermore, writing it down will allow you to free it from working memory.

Plan your quitting time is important.

Why You Should Care About Compilers

From this blog post:

1. Computing on edge devices can save you tons of money (and potentially complexity).
2. Re-writing models to work on edge devices is costly b/c that requires specialists to execute – good compilers can make this easy or obviate that transition by helping you translate your high-level framework code into hardware-specific code.
3. Sometimes, cloud computing is not an option (you have to run on a device).
4. New types of architectures can perform very poorly on hardware without the right compilers, even when those models are significantly smaller and simpler than established models. One example is shared by Google researchers who tried to implement capsule networks.
5. Translating framework code to various backend hardware is a non-trivial problem, as illustrated in the below graph (growing number of frameworks + hardware/devices):

Terminology

It’s easy to get lost in the terminology of compilers. Here are some terms that are important to understand:

• Kernel Fusion: this is the process of combining multiple operations into a single operation. For example, a convolution followed by a ReLU can be combined into a single operation.
• Intermediate Representation (IR): The IR is a representation of a model that’s independent of a framework like Pytorch. These IRs are often JSON, YAML, or a string that encodes model structure, weights, and so on. These IRs are eventually translated into programs that can be executed and are often optimized toward specific hardware.
• Graph Lowering: This is the process of translating one IR to another. For example, a model represented in the ONNX IR can be lowered to a model in TensorRT IR. The reason for doing this is to translate one IR to one that allows a specific compiler to perform domain/hardware-specific optimizations. Different compilers can optimize different types of things and offer different goals. For example, ONNX’s primary goal is to be a portable IR, while TensorRT’s primary goal is to optimize for inference. The reason it’s called Graph lowering is that a model be represented as a graph, and you “lower” the graph representation consecutively towards something that can be made into machine code. Compilers can process IRs in different phases such that the IR is simplified/optimized, that a back-end will use to generate machine code.
• Front-End: This refers to part of the Compiler Stack that translates a programming language into an IR.
• Back-End: This refers to part of the Compiler Stack that translates an IR into machine code. You can compose different front-ends with different back-ends through “middle-ends” that translate IRs from one to another (also known as “lowering” or “graph lowering”).
• Graph Breaks: When you represent a model as a graph via an intermediate representation (IR) there are pieces of code that might not fit into a graph form, like if/else statements. When this happens, you will have a graph break, where you will have to split your program into several graphs. Only the parts that can be represented as a graph can usually be optimized by the compiler. Depending on the framework you are using, either your or the framework will have to stitch all the graphs together in a way that represents the original program. Graph breaks usually incur a performance penalty, so it’s important to minimize them.

The Compiler Stack

It can often be ambiguous when someone refers to a compiler if they are referring to a front-end that generates the IR, the back-end that generates/executes the code, or the entire compiler stack as a whole. For example, ONNX has both an intermediate representation IR and a runtime. The IR is a specification (a string) that allows you to represent a model in a way that’s independent of a framework. The ONNX Runtime is a backend that allows you to execute models represented by the ONNX IR on a variety of hardware and from various languages (Python, C++, Java, JS, etc.).

The term “compiler” is often overloaded. Understanding the context in which the term is used can be helpful for understanding documentation. For example, Torch Dynamo is a front-end (often referred to as a “graph acquisition tool”) that produces an IR. The user can then lower this IR to a back-end to another compiler stack like C++ (for CPUs) or OpenAI Triton (for GPUs) that eventually gets executed.

Nvidia vs OpenAI Triton

Triton by Nvidia is an inference server. Triton by OpenAI is a high-level CUDA programming language and compiler stack. The two are not related.

Training: TorchDyamo + JIT + Triton

Compilers are often thought of as optimizations for inference. However, there are compilers that help with training too. The most notable these days is TorchDynamo + JIT Compiler. TorchDynamo is a front-end that allows you to capture a PyTorch model as an IR¹. Since this front-end is maintained by the Pytorch team, it is going to have the greatest support for Pytorch models. The best thing about the TorchDynamo+JIT stack is that it seamlessly handles graph breaks for you (stitches various subgraphs together for you, etc). The JIT compiler supplants the CPython interpreter that normally “eagerly” runs your PyTorch code for faster execution². It works by dynamically modifying Python bytecode right before it is executed. All you have to do is to call one line of code: torch.compile(...) to see the benefits. Andrej Karpathy is using this in his NanoGPT tutorials:

This is so underrated! Ridiculous speedup by calling torch.compile(...)

Code: https://t.co/9mMGGyEcfz

Related blog posts:
1. https://t.co/JOLZ8BQdet
2. https://t.co/ci8KAzLW1p (this blog by @marksaroufim is a real gem, BTW) https://t.co/E0oRVC4Ow8

— Hamel Husain (@HamelHusain) February 7, 2023

The JIT Compiler can theoretically leverage different backends for execution, but at this time, the paved path is to use the TorchInductor which lowers the IR to the OpenAI Triton backend. So the entire “compiler stack” looks like this:

1. TorchDynamo acquires a PyTorch model as an IR (FX graphs, multiple graphs if there are graph breaks)
2. The JIT Compiler lowers the FX Graph (through other intermediate compiler stacks) to TorchInductor
3. TorchInductor lowers the IR to OpenAI-Triton for GPU or C++ for CPU
4. OpenAI Triton compiles the IR and executes it (perhaps by first passing it to some other back-end like CUDA)

That is a lot of steps! While the end-user doesn’t necessarily need to be aware of all these steps, the documentation on Torch Dynamo can be really confusing unless you are aware of these different things.

TensorRT

This is a compiler/execution environment. Unlike the JIT (just-in-time), it is an AOT (ahead-of-time) compiler. It is compatible with PyTorch via torch-TensorRT. From the docs: TensorRT’s primary means of importing a trained model from a framework is through the ONNX interchange format, so you need to convert your model to ONNX before getting to TensorRT (this happens automatically when you use torch-TensorRT).

ONNX

ONNX is mainly focused on portability. ONNX provides its own set of ops that are cross-platform and can be run on an ONNX Runtime. The ONNX runtime provides clients in many languages, like Python, C, C++, Java, JS, etc., allowing you to load a model for either inferencing or training. There are hardware-specific clients that are optimized for OS (i.e., Linux, Windows, Mac, etc.), Hardware acceleration (CUDA, CoreML) and so forth. You can select a client from here: https://onnxruntime.ai/.

You can construct your computation graphs with ONNX’s built-in ops; However, this is an impractical way to build a model. In practice, you want to use your favorite ML framework like PyTorch or Tensorflow and use a converter to convert that model to ONNX like torch.onnx

OpenAI Triton

Only for GPUs, from this talk:

ApacheTVM

ApacheTVM is a compiler that seems to have one of the widest ranges of hardware support.

Adaptive Compilers (TVM)

autoTVM, part of ApacheTVM compiler framework are part of a class of compilers that are adaptive that run tests on hardware and automatically try to optimize it for the hardware by trying many different graph configurations. From Chip’s intro to compilers:

autoTVM measures the actual time it takes to run each path it goes down, which gives it ground truth data to train a cost model to predict how long a future path will take. The pro of this approach is that because the model is trained using the data generated during runtime, it can adapt to any type of hardware it runs on. The con is that it takes more time for the cost model to start improving.

One example is when Apple released their M1 chips in Nov 2020. M1 is an ARM-based system on a chip, and ARM architectures are more or less well-understood. However, M1 still has a lot of novel components of its ARM implementation and requires significant optimization to make various ML models run fast on it. A month after the release, folks at OctoML showed that the optimization made by autoTVM is almost 30% faster than hand-designed optimization by Apple’s Core ML team.

Quote from a compiler expert:

Pros of dynamic compilers is that we maximize performance across a broader set of models/ops. Downside of dynamic compilers is search time. AutoTVM generally takes hours, even more than a day to compile because it’s trying every possible combination.

ML can also be used to optimize the computation graph, or “ML is optimizing ML”. Below is an illustration from Chen, et.al; TVM: An Automated End-to-End Optimizing Compiler for Deep Learning, where they comment that:

Possible optimizations form a large space, so we use an ML-based cost model to find optimized operators.

Adaptive compilers are interesting because they do not depend on hand-coded rules like traditional compilers and seem more user-friendly and generalizable to new kinds of hardware. Another adaptive compiler is cuDNN autotune, which you can enable with torch.backends.cudnn.benchmark=True, but apparently this only works with convolutions. The drawback of adaptive compilers is that they can take a long time to “learn” or “search” for good optimizations. However, you can amortize the cost of optimizing the model over many devices on that hardware, and can even take that checkpoint as a starting point for future tuning sessions.

I find adaptive compilation more exciting and promising as a future direction in compilers.

WASM

Targeting WASM would theoretically enable you to work on any device that can run a browser - and many devices/hardware can run browsers! However, one drawback is that it’s still slow. See this blog post for a more detailed discussion.

Use Playhead as A Cutting Tool

Youtube Tutorial: https://www.youtube.com/watch?v=yh77878QDVE His playlist: https://www.youtube.com/playlist?list=PLL6tMzF36ox2c–SNKiifuP8kEFh80wPu

CMD + B -> “Blade” CMD + SHIFT + [ or ] to cut to location

Talking Head Circle Thingy

Here is a circular camera filter with OBS, which might be easier than DVR.

You can crop like this

Pause

You can add pause recording as a hotkey in OBS

Source freshness

The freshness config can be added to models/staging/jaffle_shop/src_jaffle_shop.yml, and must reference a field in the table via loaded_at_field:

version: 2

sources:
  - name: jaffle_shop
    database: dbt-tutorial
    schema: jaffle_shop
    tables:
      - name: customers
      - name: orders
        loaded_at_field: _etl_loaded_at
        freshness:
          warn_after: {count: 12, period: hour}
          error_after: {count: 24, period: hour}

In the above example, the freshness applies to the orders table. You can run the dbt source freshness command to see the freshness of your sources:

> dbt source freshness
...
17:52:22  1 of 1 WARN freshness of jaffle_shop.orders .................................... [WARN in 1.32s]
17:52:22  Finished 

Generic Tests

We can add a unique test to models/staging/jaffle_shop/stg_jaffe_shop.sql:

version: 2

models:
  - name: stg_customers
    columns: 
      - name: customer_id
        tests:
          - unique
          - not_null
  - name: stg_orders
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: status
        tests:
          - accepted_values:
              values:
                - completed
                - shipped
                - returned
                - return_pending
                - placed

You can run the test by calling

dbt test

You can also test a specific model by calling:

dbt test --models stg_customers

Singular Tests

We can add a test in tests/assert_positive_total_for_payments.sql:

with payments as (
    select * from {{ ref('stg_payments') }}
)

select
    order_id,
    sum(amount) as total_amount
from payments
group by order_id
having not(total_amount < 0)

select
      count(*) as failures,
      count(*) != 0 as should_warn,
      count(*) != 0 as should_error
...

It knows that this file is a test because of the configuration at the root of the project dbt_project.yml:

test-paths: ["tests"]
...

Testing Sources

You can test sources in the yaml file that configures the source, or as a separate file in the test path. If we want to add a test to models/staging/jaffle_shop/src_jaffle_shop.yml, we can add the following:

version: 2

sources:
  - name: jaffle_shop
    database: dbt-tutorial
    schema: jaffle_shop
    tables:
      - name: customers
        columns:
          - name: id
            tests:
              - unique
              - not_null
            
      - name: orders
        columns:
          - name: id
            tests:
              - unique              
              - not_null
        loaded_at_field: _etl_loaded_at
        freshness:
          warn_after: {count: 12, period: hour}
          error_after: {count: 24, period: hour}

We can execute this test with dbt test or just test sources with the command: dbt test --select source:jaffle_shop.

Generating Documentation

You have to run the command dbt docs generate to re-generate the docs. You can then find the descriptions you added on the docs page:

The doc block is rendered as well:

Blocks

They also have a very convenient lambda like thing called Blocks:

sum = 0 
[4,6,8].each { |x| sum += x 
               puts sum }

You can use Blocks to make accumulators too, and even use inject to initialize the accumulator:

sum = [4,6,8].inject(0) { |acc,elt| acc + elt }

To use blocks in a method, you will have to look that up in the docs. This involves the yield keyword. For example, this code will print “hi” 3 times:

def foo x 
  if x 
    yield 
   else 
    yield 
    yield 
   end 
end 

foo (true) { puts "hi" } 
foo (false) { puts "hi" }

Blocks are not first class functions even though they kind of look like lambdas. Lets say you wanted to map over an array but wanted to return an array of functions instead of values. The way to do this is to use the keyword lambda:

c = a.map {|x| {|y| x >= y} } # wrong, a syntax error

c = a.map {|x| lambda {|y| x >= y} } # this will work

class Child < Parent
 ...
end

Typing

They discussed the various ways different type systems are constructed. The interface idiom, that is familar to you from Golang (but not specific to Golang) was introduced.