Dep Tutorial

Getting started
1. Create a new folder called depSandbox, and in there, another folder called foo. Here's how:
```
$ rm -ir depSandbox
$ mkdir -p depSandbox/foo
$ cd depSandbox/foo
```
2. In that folder, write a DepFile containing:
```
include /etc/depsyntax

cc -c -o foo.o foo.c
cc -o foo foo.o
```
3. Run dep. It should fail with the error message:
```
(dep: Can't read cache file .DepFileCache)
Targetting foo
foo.c (needed for foo.o) does not exist and has no write rule
```
4. Write foo.c containing:
```
#include <stdio.h>

int main() {
  printf("Hello from foo\n");
  return 0;
}
```
5. Run dep. This time it should say:
```
(dep: Can't read cache file .DepFileCache)
Targetting foo
cc -c -o foo.o foo.c
cc -o foo foo.o
```
  This is in one sense unsurprising: those are the commands given in the DepFile. But, being a build tool, dep should be following file dependencies. Where are the dependencies?
  
  This is also unsurprising to anyone who is familiar with C: The input is foo.c, the intermediate is foo.o, and the output is foo. This is all very clear from the commands.
  
  dep determines the inputs and outputs by analysing the build command. To do so, it uses a hint in the file /etc/depsyntax, which is the first thing it included and read. One line of that file is:
```
syntax -mixed cc -g -f: -I:%includedir -W: -o:%out \
  -c%findincludes -l:%library -L:%libdir -.* %in
```
  Among other things, this tells dep that cc's -o option indicates an output file (%out), and that all non-option parameters are input files (%in).
  
  So you don't actually have to specify the inputs and outputs of a command. dep can analyse the command and figure them out for itself.
  
  (Some commands aren't quite so clear as to what their inputs or (especially) outputs are; more on those later.)
6. Run ./foo. It should say:
```
Hello from foo
```
Include files and multiple object files
1. Add some lines to foo.c (specifically the "bar" lines):
```
#include <stdio.h>
#include "bar.h"

int main() {
  printf("Hello from foo\\n");
  printf("bar() says %d\\n", bar());
  return 0;
}
```
2. Add bar.h:
```
#ifndef BAR_H
#define BAR_H

int bar();

#endif
```
3. And add bar.c:
```
#include "bar.h"

int bar() {
  return 17;
}
```
4. Edit DepFile and add in the new source and object files:
```
include /etc/depsyntax
cc -c -o foo.o foo.c
cc -c -o bar.o bar.c
cc -o foo foo.o bar.o
```
5. And run dep --fixedorder . (The --fixedorder option cuts build times out of the job sorting algorithm. Normally dep starts time-consuming build steps sooner, to minimise the total build time. But this document is also a test script, and the more deterministic it is, the better it works.) It should say:
```
Targetting foo
cc -c -o bar.o bar.c
cc -c -o foo.o foo.c
cc -o foo foo.o bar.o
```
6. Run dep again. It won't do anything:
```
Targetting foo
dep: all targets are up to date
```
7. Touch bar.h:
```
$ touch bar.h
```
8. And run dep again. This time it should say:
```
Targetting foo
cc -c -o bar.o bar.c
cc -c -o foo.o foo.c
cc -o foo foo.o bar.o
```
  Note that dep recompiled and relinked those programs due to the touched file bar.h, which is not mentioned in the DepFile. That is, it automatically included bar.h in its file dependency graph.
  
  (make can do this, with some extra include directives, and the help of a compiler option that can output make's dependency syntax. (The tool ninja can also read make's dependency syntax. (Which I find somewhat strange. (Please excuse all these brackets.))))
  
  dep takes a simpler approach: it reads the #include directives directly from the source files. Note that dep doesn't have a C++ parser hard-coded in; instead, it is in the configuration file /etc/depsyntax, in this line:
```
findincludes '#include[ \t]+"(.*)"[ \t]*\r?' *.c *.cpp *.h
```
  It also does not read the entire file, just the first thousand lines (also configurable), and it is not reading the includes every time, it only reads them when the files have changed; when they haven't changed, it instead uses a copy of the includes in its cache file, in lines that read:
```
fileinfo bar.c OK 2024-05-30 17:46:04.8950837+10 47
  failtime 1970-01-01 10:00:00+10 buildseconds 0 includes bar.h
fileinfo foo.c OK 2024-05-30 17:46:05.6403056+10 188
  failtime 1970-01-01 10:00:00+10 buildseconds 0 includes bar.h ../quux/quux.h
```
  Along with other information, such as: old modification time, old file size, and a few other things.
  
  Also, whereas make internally models include files as extra source files, dep internally models them as actual include files. This more realistic modelling saves a bit of time and space on big projects. Consider a project with ten .c files which all include one .h file that in turn includes ten other .h files: In dep, that requires 30 file relationships; in make, it is 110.
9. One more thing: run ./foo. It should say:
```
Hello from foo
bar() says 17
```
Patterns and rebuilding
1. When there are a lot of commands that differ only by a couple of filenames, you can use a pattern rule. In such a rule, the % symbol represents a string that can have any value. Change DepFile to:
```
include /etc/depsyntax
cc -c -o %.o %.c
cc -o foo foo.o bar.o
```
2. And run dep -a --fixedorder . (The -a option causes a full rebuild.) This time dep should say:
```
Targetting foo
cc -c -o foo.o foo.c
cc -c -o bar.o bar.c
cc -o foo foo.o bar.o
```
Libraries and folders
1. Add some more lines to foo.c (specifically the "quux" lines):
```
#include <stdio.h>
#include "bar.h"
#include "quux.h"

int main() {
  printf("Hello from foo\\n");
  printf("bar() says %d\\n", bar());
  printf("quux() says %d\\n", quux());
  return 0;
}
```
2. Add a new folder called ../quux. (In dep style, the folders of libraries and apps form a shallow hierachy of siblings instead of a deep hierarchy of dependencies):
```
$ mkdir ../quux
$ cd ../quux
```
3. In there, add quux.h:
```
#ifndef QUUX_H
#define QUUX_H

int quux();

#endif
```
4. And add quux.c:
```
#include "quux.h"

int quux() {
  return 123;
}
```
5. And add a DepFile for a shared library:
```
include /etc/depsyntax
cc -c -o quux.o quux.c
mkdir -p ../bin
cc -shared -o ../bin/libquux.so quux.o
```
6. cd back to ../foo:
```
$ cd ../foo
```
  and edit the DepFile:
```
include ../quux/DepFile
cc -c -o foo.o -I . -I ../quux foo.c
cc -c -o bar.o bar.c
mkdir ../bin
cc -o ../bin/foo foo.o bar.o -L ../bin -Wl,-rpath,../bin -l quux
```
7. Run dep --mkdir --fixedorder:
```
Targetting ../bin/foo
(in ../quux) cc -c -o quux.o quux.c
(in ../quux) mkdir -p ../bin
(in ../quux) cc -shared -o ../bin/libquux.so quux.o
cc -c -o foo.o -I . -I ../quux foo.c
cc -o ../bin/foo foo.o bar.o -L ../bin -Wl,-rpath,../bin -l quux
```
  The --mkdir option causes dep to automatically add existence dependencies between files mkdir commands. Without the option, dep does not add such dependencies, as there are a lot of them, and they are rarely needed.
  
  An existence dependency is a dependency where a target is considered up to date if its source exists. Timestamps and file sizes are not compared any further. Make calls these order-only dependencies.
8. run ../bin/foo
```
Hello from foo
bar() says 17
quux() says 123
```
9. Edit ../quux/quux.c and change the return value of the function:
```
#include "quux.h"

int quux() {
  return 4321;
}
```
10. And run dep again:
```
Targetting ../bin/foo
(in ../quux) cc -c -o quux.o quux.c
(in ../quux) cc -shared -o ../bin/libquux.so quux.o
```
11. It is worth noting here is that dep did not re-link the final executable ../bin/foo. Most of the time, it doesn't have to: final linkage is actually done by the loader. To be sure that the program still works, run ../bin/foo:
```
Hello from foo
bar() says 17
quux() says 4321
```
  These are existence dependencies again. /etc/depsyntax has the line:
```
exists_file_ext .so .dll
```
  This means that any source file with the extension .so or .dll gets an existence dependency instead of of an ordinary source dependency.
12. On occasion, you might prefer (or need) to do a full relink that does not skip any steps. Do do this, run dep --noskip:
```
Targetting ../bin/foo
cc -o ../bin/foo foo.o bar.o -L ../bin -Wl,-rpath,../bin -l quux
```
The --sense option
1. dep typically runs a command when:
  - one of its source files is newer than one of its target files
  - the build command for a target file has changed
  But there are other possible triggers for running a command.
2. With --sense=modtime, any change to a source file's modification time will trigger the corresponding command. Here's how:
```
$ touch -t2312311655 foo.c
```
3. Run dep. It will simply say:
```
Targetting ../bin/foo
dep: all targets are up to date
```
4. Run dep --sense=modtime. This time it will say:
```
Targetting ../bin/foo
cc -c -o foo.o -I . -I ../quux foo.c
cc -o ../bin/foo foo.o bar.o -L ../bin -Wl,-rpath,../bin -l quux
```
5. With --sense=size, any change to a source file's size will also trigger the corresponding command. To set this up:
```
$ echo >> foo.c
$ touch -t2312311655 foo.c
```
6. Run dep --sense=newer,rule,modtime. It will simply say:
```
Targetting ../bin/foo
dep: all targets are up to date
```
7. Run dep --sense=newer,rule,modtime,size (or equivalently, dep --sense=all). This time it will say:
```
Targetting ../bin/foo
cc -c -o foo.o -I . -I ../quux foo.c
cc -o ../bin/foo foo.o bar.o -L ../bin -Wl,-rpath,../bin -l quux
```
  (dep --sense=all is a shortcut for --sense=newer,rule,modtime,size)

%cmpout

Some programs produce output files that change so rarely, it is better to not just assume they have changed, but to check them instead.
Bison is such a program. It can produce two output files: (1) a generated parser, and (2) a header file that contains token values for interfacing with such things as a lexical scanner, and parse tree- or AST-related code. The header file only changes when the list of tokens change, which compared to the rest of the parser, is not that often. So it makes sense to check that the file really changed, before recompiling its dependents.

In dep, this type of output file is known as a %cmpout file and /etc/depsyntax has one in its syntax for bison:
```
syntax bison -W: -t -v -o:%out --defines=%cmpout %in
```
To see this in action, create a minimal bison file, intercalparse.y:
```
%define api.prefix {intercal}
%{
#include <stdio.h>
int intercallex();
void intercalerror(const char *error) {
  fprintf(stderr, "%s\n", error);
}
%}
%token KEYWORD_INTERLEAVE
%%
filecontents: KEYWORD_INTERLEAVE {
}
```
Add intercallex() and a #include statement for intercalparse.h to foo.c:
```
#include <stdio.h>
#include "bar.h"
#include "quux.h"
#include "intercalparse.h"

int main() {
  printf("Hello from foo\\n");
  printf("bar() says %d\\n", bar());
  printf("quux() says %d\\n", quux());
  return 0;
}

int intercallex() {
  return KEYWORD_INTERLEAVE;
}
```
Add DepFile commands to generate and compile intercalparse.c:
```
include ../quux/DepFile
bison -o intercalparse.c --defines=intercalparse.h intercalparse.y
cc -c -o intercalparse.o intercalparse.c
cc -c -o foo.o -I . -I ../quux foo.c
cc -c -o bar.o bar.c
mkdir ../bin
cc -o ../bin/foo foo.o bar.o intercalparse.o -L ../bin -l quux
```

Run dep. It will generate, compile link the new file, and also note that intercalparse.h changed:

Targetting ../bin/foo
bison -o intercalparse.c --defines=intercalparse.h intercalparse.y
intercalparse.h changed
cc -c -o foo.o -I . -I ../quux foo.c
cc -c -o intercalparse.o intercalparse.c
cc -o ../bin/foo foo.o bar.o intercalparse.o -L ../bin -l quux

Touch intercalparse.y:
```
$ touch intercalparse.y
```

Run dep again. This time, dep will notice that intercalparse.h did not change, and will not recompile the file that includes it (foo.c):

Targetting ../bin/foo
bison -o intercalparse.c --defines=intercalparse.h intercalparse.y
intercalparse.h did not change
cc -c -o intercalparse.o intercalparse.c
cc -o ../bin/foo foo.o bar.o intercalparse.o -L ../bin -l quux

Run dep again. This time there will be nothing to do:

Targetting ../bin/foo
dep: all targets are up to date

Things get tricky when you split this behavior over multiple invocations of dep. Touch intercalparse.y again:
```
$ touch intercalparse.y
```

And run dep intercalparse.o. dep will notice that intercalparse.h did not change, and will regenerate and recompile intercalparse.c:

bison -o intercalparse.c --defines=intercalparse.h intercalparse.y
intercalparse.h did not change
cc -c -o intercalparse.o intercalparse.c

Run dep. It should just link without recompiling foo.c:

Targetting ../bin/foo
cc -o ../bin/foo foo.o bar.o intercalparse.o -L ../bin -l quux

Run dep again. This time there will be absolutely nothing to do:
```
Targetting ../bin/foo
dep: all targets are up to date
```