Main

1. Introduction

This file contains the source code for main.d the file which contains the main function for Literate. This will parse any arguments, show help text and finally run tangle or weave (or both) on any input files.

Here is an overview:

{main.d 1}

{Imports, 5}

{getLinenums function, 5}
{lit function, 3}

void main(in string[] args) {
    string[] files = [];
    {Parse the arguments, 2}
    {Run Literate, 3}
}

2. Parsing the Arguments

The arguments will consist of either flags or input files. The flags Literate accepts are:

--help -h Show the help text
--tangle -t Only run tangle
--weave -w Only run weave
--no-output Do not generate any output
--out-dir -odir DIR Put the generated files in DIR
--compiler Don't ignore the @compiler command
--linenums -l STR Write line numbers prepended with STR to the output file
--version Show the version number and compiler information

All other inputs are input files.

We also need some variables to store these flags in, and they should be global so that the rest of the program can access them.

{Globals 2}

bool tangleOnly;
bool isBook;
bool weaveOnly;
bool noOutput;
bool noCompCmd = true;
bool tangleErrors;
bool useStdin;
bool lineDirectives;
string versionNum = "0.1";
string outDir = "."; // Default is current directory
string lineDirectiveStr;

string helpText =
"Lit: Literate Programming System\n"
"\n"
"Usage: lit [options] <inputs>\n"
"\n"
"Options:\n"
"--help       -h         Show this help text\n"
"--tangle     -t         Only compile code files\n"
"--weave      -w         Only compile HTML files\n"
"--no-output  -no        Do not generate any output files\n"
"--out-dir    -odir DIR  Put the generated files in DIR\n"
"--compiler   -c         Report compiler errors (needs @compiler to be defined)\n"
"--linenums   -l    STR  Write line numbers prepended with STR to the output file\n"
"--version    -v         Show the version number and compiler information";

This program uses a number of block modifiers in order to facilitate certain functionality. i.e. If you don't wish a code block to be woven into the final HTML then the noWeave modifier will indicate this for you.

Each modifier is represented by this list of enums:

{Modifiers 2}

enum Modifier {
    noWeave,
    noTangle,
    additive, // +=
    redef // :=
}

We'll put these two blocks in their own file for "globals".

{globals.d 2}

{Globals, 2}
{Modifiers, 2}

Now, to actually parse the arguments:

{Parse the arguments 2}

for (int i = 1; i < args.length; i++) {
    auto arg = args[i];
    if (arg == "--help" || arg == "-h") {
        writeln(helpText);
        return;
    } else if (arg == "--tangle" || arg == "-t") {
        tangleOnly = true;
    } else if (arg == "--weave" || arg == "-w") {
        weaveOnly = true;
    } else if (arg == "--no-output" || arg == "-no") {
        noOutput = true;
    } else if (arg == "--out-dir" || arg == "-odir") {
        if (i == args.length - 1) {
            writeln("No output directory provided.");
            return;
        }
        outDir = args[++i];
    } else if (arg == "--compiler" || arg == "-c") {
        noCompCmd = false;
        noOutput = true;
    } else if (arg == "--linenums" || arg == "-l") {
        lineDirectives = true;
        if (i == args.length - 1) {
            writeln("No line number string provided.");
            return;
        }
        lineDirectiveStr = args[++i];
    } else if (arg == "--version" || arg == "-v") {
        writeln("Literate version " ~ versionNum);
        writeln("Compiled by " ~ __VENDOR__ ~ " on " ~ __DATE__);
        return;
    } else if (arg == "-") {
        useStdin = true;
    } else {
        files ~= arg;
    }
}

Used in section 1

3. Run Literate

To run literate we go through every file that was passed in, check if it exists, and run tangle and weave on it (unless tangleOnly or weaveOnly was specified).

{Run Literate 3}

if (files.length > 0) {
    foreach (filename; files) {
        if (!filename.exists()) {
            writeln("File ", filename, " does not exist!");
            continue;
        }
        File f = File(filename);
        string fileSrc = readall(f);

        lit(filename, fileSrc);
    }
} else if (useStdin) {
    string stdinSrc = readall();
    lit("stdin", stdinSrc);
} else  {
    writeln(helpText);
}

Used in section 1

The lit function parses the text that is inputted and then either tangles, weaves, or both. Finally it Checks for compiler errors if the --compiler flag was passed.

{lit function 3}

void lit(string filename, string fileSrc) {
    Program p = new Program();
    p.file = filename;
    if (fileSrc.matchFirst("(\n|^)@book\\s*?(\n|$)")) {
        isBook = true;
        p = parseProgram(p, fileSrc);
        if (p.chapters.length == 0) {
            error(filename, 1, "This book has no chapters");
            return;
        }
    } else {
        Chapter c = new Chapter();
        c.file = filename;
        c.majorNum = 1; c.minorNum = 0;

        c = parseChapter(c, fileSrc);
        p.chapters ~= c;
    }

    if (!weaveOnly) {
        tangle(p);
    }

    if (!tangleOnly) {
        weave(p);
    }

    if (!noCompCmd && !tangleErrors && !weaveOnly) {
        {Check for compiler errors, 4}
    }
}

Used in section 1

4. Compiler errors

Here we check for compiler errors.

First we have to get all the codeblocks so that we can backtrack the line numbers from the error message to the correct codeblock. Then we can use the getLinenums function to get the line numbers for each line in the tangled code.

{Check for compiler errors 4}

Line[][string] codeLinenums;

Block[string] rootCodeblocks;
Block[string] codeblocks;
getCodeblocks(p, codeblocks, rootCodeblocks);

foreach (b; rootCodeblocks) {
    codeLinenums = getLinenums(codeblocks, b.name, b.name, codeLinenums);
}

Added to in sections 4 and 4

Used in section 3

Now we go and check for the @compiler command and the @error_format command.

{Check for compiler errors 4} +=

string compilerCmd;
string errorFormat;
Command errorFormatCmd;
foreach (cmd; p.commands) {
    if (cmd.name == "@compiler") {
        compilerCmd = cmd.args;
    } else if (cmd.name == "@error_format") {
        errorFormat = cmd.args;
        errorFormatCmd = cmd;
    }
}
if (p.chapters.length == 1) {
    Chapter c = p.chapters[0];
    foreach (cmd; c.commands) {
        if (cmd.name == "@compiler") {
            compilerCmd = cmd.args;
        } else if (cmd.name == "@error_format") {
            errorFormat = cmd.args;
            errorFormatCmd = cmd;
        }
    }
}

Added to in sections 4 and 4

Used in section 3

If there is no @error_format but the @compiler command uses a known compiler, we can substitute the error format in.

Supported compilers/linters are:

clang
gcc
g++
javac
pyflakes
jshint
dmd

{Check for compiler errors 4} +=

if (errorFormat is null) {
    if (compilerCmd.indexOf("clang") != -1) { errorFormat = "%f:%l:%s: %s: %m"; }
    else if (compilerCmd.indexOf("gcc") != -1) { errorFormat = "%f:%l:%s: %s: %m"; }
    else if (compilerCmd.indexOf("g++") != -1) { errorFormat = "%f:%l:%s: %s: %m"; }
    else if (compilerCmd.indexOf("javac") != -1) { errorFormat = "%f:%l: %s: %m"; }
    else if (compilerCmd.indexOf("pyflakes") != -1) { errorFormat = "%f:%l:(%s:)? %m"; }
    else if (compilerCmd.indexOf("jshint") != -1) { errorFormat = "%f: line %l,%s, %m"; }
    else if (compilerCmd.indexOf("dmd") != -1) { errorFormat = "%f\\(%l\\):%s: %m"; }
}

Added to in sections 4 and 4

Used in section 3

Now we actually go through and create the regex, by replacing the %l, %f, and %m with matched regular expressions. Then we execute the shell command, parse each error using the error format, and rewrite the error with the proper filename and line number given by the array codeLinenums that we created earlier.

{Check for compiler errors 4} +=

if (errorFormat !is null) {
    if (errorFormat.indexOf("%l") != -1 && errorFormat.indexOf("%f") != -1 && errorFormat.indexOf("%m") != -1) {
        auto r = regex("");
        try {
            r = regex("^" ~ errorFormat.replaceAll(regex("%s"), ".*?")
                                                   .replaceAll(regex("%l"), "(?P<linenum>\\d+?)")
                                                   .replaceAll(regex("%f"), "(?P<filename>.*?)")
                                                   .replaceAll(regex("%m"), "(?P<message>.*?)") ~ "$");
        } catch (Exception e) {
            error(errorFormatCmd.filename, errorFormatCmd.lineNum, "Regular expression error: " ~ e.msg);
            return;
        }

        writeln(compilerCmd);
        auto output = executeShell(compilerCmd).output.split("\n");
        int i = 0;

        foreach (line; output) {
            auto matches = matchFirst(line, r);

            string linenum = matches["linenum"];
            string fname = matches["filename"];
            string message = matches["message"];

            if (linenum != "" && fname != "") {
                if (codeLinenums[fname].length > to!int(linenum)) {
                    auto codeline = codeLinenums[fname][to!int(linenum) - 1];
                    error(codeline.file, codeline.lineNum, message);
                } else {
                    auto codeline = codeLinenums[fname][codeLinenums[fname].length - 2];
                    error(codeline.file, codeline.lineNum, message);
                }
            } else {
                if (!(line == "" && i == output.length - 1)) {
                    writeln(line);
                }
            }
            i++;
        }
    }
}

Added to in sections 4 and 4

Used in section 3

5. GetLinenums

Here is the getLinenums function. It just goes through every block like tangle would, but for each line it adds the line to the array, storing the file and line number for that line.

{getLinenums function 5}

Line[][string] getLinenums(Block[string] codeblocks, string blockName, 
                 string rootName, Line[][string] codeLinenums) {
    Block block = codeblocks[blockName];

    if (block.commentString != "") {
        codeLinenums[rootName] ~= new Line("comment", "", 0);
    }

    foreach (lineObj; block.lines) {
        string line = lineObj.text;
        string stripLine = strip(line);
        if (stripLine.startsWith("@{") && stripLine.endsWith("}")) {
            auto index = stripLine.length - 1;
            auto newBlockName = stripLine[2..index];
            getLinenums(codeblocks, newBlockName, rootName, codeLinenums);
        } else {
            codeLinenums[rootName] ~= lineObj;
        }
    }
    codeLinenums[rootName] ~= new Line("", "", 0);

    return codeLinenums;
}

Used in section 1

Finally, we also have to add the imports.

{Imports 5}

import parser;
import tangler;
import weaver;
import util;
import globals;
import std.stdio;
import std.file;
import std.string;
import std.process;
import std.regex;
import std.conv;

Used in section 1