|
|
<!DOCTYPE html> |
|
|
<html lang="en"><head><meta charset="UTF-8"/><meta name="viewport" content="width=device-width, initial-scale=1.0"/><title>3. Processing pipeline · Literate.jl</title><link href="https://cdnjs.cloudflare.com/ajax/libs/normalize/4.2.0/normalize.min.css" rel="stylesheet" type="text/css"/><link href="https://fonts.googleapis.com/css?family=Lato|Roboto+Mono" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.6.3/css/font-awesome.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css" rel="stylesheet" type="text/css"/><script>documenterBaseURL="."</script><script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.2.0/require.min.js" data-main="assets/documenter.js"></script><script src="siteinfo.js"></script><script src="../versions.js"></script><link href="assets/documenter.css" rel="stylesheet" type="text/css"/></head><body><nav class="toc"><h1>Literate.jl</h1><select id="version-selector" onChange="window.location.href=this.value" style="visibility: hidden"></select><form class="search" id="search-form" action="search.html"><input id="search-query" name="q" type="text" placeholder="Search docs"/></form><ul><li><a class="toctext" href="index.html"><strong>1.</strong> Introduction</a></li><li><a class="toctext" href="fileformat.html"><strong>2.</strong> File Format</a></li><li class="current"><a class="toctext" href="pipeline.html"><strong>3.</strong> Processing pipeline</a><ul class="internal"><li><a class="toctext" href="#Pre-processing-1"><strong>3.1.</strong> Pre-processing</a></li><li><a class="toctext" href="#Parsing-1"><strong>3.2.</strong> Parsing</a></li><li><a class="toctext" href="#Document-generation-1"><strong>3.3.</strong> Document generation</a></li><li><a class="toctext" href="#Post-processing-1"><strong>3.4.</strong> Post-processing</a></li><li><a class="toctext" href="#Writing-to-file-1"><strong>3.5.</strong> Writing to file</a></li></ul></li><li><a class="toctext" href="outputformats.html"><strong>4.</strong> Output Formats</a></li><li><a class="toctext" href="customprocessing.html"><strong>5.</strong> Custom pre- and post-processing</a></li><li><a class="toctext" href="documenter.html"><strong>6.</strong> Interaction with Documenter.jl</a></li><li><a class="toctext" href="generated/example.html"><strong>7.</strong> Example</a></li></ul></nav><article id="docs"><header><nav><ul><li><a href="pipeline.html"><strong>3.</strong> Processing pipeline</a></li></ul><a class="edit-page" href="https://github.com/fredrikekre/Literate.jl/blob/master/docs/src/pipeline.md"><span class="fa"></span> Edit on GitHub</a></nav><hr/><div id="topbar"><span>3. Processing pipeline</span><a class="fa fa-bars" href="#"></a></div></header><h1><a class="nav-anchor" id="**3.**-Processing-pipeline-1" href="#**3.**-Processing-pipeline-1"><strong>3.</strong> Processing pipeline</a></h1><p>The generation of output follows the same pipeline for all output formats:</p><ol><li><p><a href="pipeline.html#Pre-processing-1">Pre-processing</a></p></li><li><p><a href="pipeline.html#Parsing-1">Parsing</a></p></li><li><p><a href="pipeline.html#Document-generation-1">Document generation</a></p></li><li><p><a href="pipeline.html#Post-processing-1">Post-processing</a></p></li><li><p><a href="pipeline.html#Writing-to-file-1">Writing to file</a></p></li></ol><h2><a class="nav-anchor" id="Pre-processing-1" href="#Pre-processing-1"><strong>3.1.</strong> Pre-processing</a></h2><p>The first step is pre-processing of the input file. The file is read to a <code>String</code>. The first processing step is to apply the user specified pre-processing function, see <a href="customprocessing.html#Custom-pre-and-post-processing-1">Custom pre- and post-processing</a>.</p><p>The next step is to perform all of the built-in default replacements. CRLF style line endings (<code>"\r\n"</code>) are replaced with LF line endings (<code>"\n"</code>) to simplify internal processing. Next, line filtering is performed, see <a href="fileformat.html#Filtering-Lines-1">Filtering Lines</a>, meaning that lines starting with <code>#md</code>, <code>#nb</code> or <code>#jl</code> are handled (either just the token itself is removed, or the full line, depending on the output target). The last pre-processing step is to expand the convenience "macros" described in <a href="fileformat.html#Default-Replacements-1">Default Replacements</a> is expanded.</p><h2><a class="nav-anchor" id="Parsing-1" href="#Parsing-1"><strong>3.2.</strong> Parsing</a></h2><p>After the preprocessing the file is parsed. The first step is to categorize each line and mark them as either markdown or code according to the rules described in the <a href="fileformat.html#Syntax-1">Syntax</a> section. Lets consider the example from the previous section with each line categorized:</p><pre><code class="language-none"># # Rational numbers <- markdown |
|
|
# <- markdown |
|
|
# In julia rational numbers can be constructed with the `//` operator. <- markdown |
|
|
# Lets define two rational numbers, `x` and `y`: <- markdown |
|
|
<- code |
|
|
x = 1 // 3 <- code |
|
|
y = 2 // 5 <- code |
|
|
<- code |
|
|
# When adding `x` and `y` together we obtain a new rational number: <- markdown |
|
|
<- code |
|
|
z = x + y <- code</code></pre><p>In the next step the lines are grouped into "chunks" of markdown and code. This is done by simply collecting adjacent lines of the same "type" into chunks:</p><pre><code class="language-none"># # Rational numbers ┐ |
|
|
# │ |
|
|
# In julia rational numbers can be constructed with the `//` operator. │ markdown |
|
|
# Lets define two rational numbers, `x` and `y`: ┘ |
|
|
┐ |
|
|
x = 1 // 3 │ |
|
|
y = 2 // 5 │ code |
|
|
┘ |
|
|
# When adding `x` and `y` together we obtain a new rational number: ] markdown |
|
|
┐ |
|
|
z = x + y ┘ code</code></pre><p>In the last parsing step all empty leading and trailing lines for each chunk are removed, but empty lines <em>within the same</em> block are kept. The leading <code>#</code> tokens are also removed from the markdown chunks. Finally we would end up with the following 4 chunks:</p><p>Chunks #1:</p><pre><code class="language-markdown"># Rational numbers |
|
|
|
|
|
In julia rational numbers can be constructed with the `//` operator. |
|
|
Lets define two rational numbers, `x` and `y`:</code></pre><p>Chunk #2:</p><pre><code class="language-julia">x = 1 // 3 |
|
|
y = 2 // 5</code></pre><p>Chunk #3:</p><pre><code class="language-markdown">When adding `x` and `y` together we obtain a new rational number:</code></pre><p>Chunk #4:</p><pre><code class="language-julia">z = x + y</code></pre><p>It is then up to the <a href="pipeline.html#Document-generation-1">Document generation</a> step to decide how these chunks should be treated.</p><h3><a class="nav-anchor" id="Custom-control-over-chunk-splits-1" href="#Custom-control-over-chunk-splits-1">Custom control over chunk splits</a></h3><p>Sometimes it is convenient to be able to manually control how the chunks are split. For example, if you want to split a block of code into two, such that they end up in two different <code>@example</code> blocks or notebook cells. The <code>#-</code> token can be used for this purpose. All lines starting with <code>#-</code> are used as "chunk-splitters":</p><pre><code class="language-julia">x = 1 // 3 |
|
|
y = 2 // 5 |
|
|
#- |
|
|
z = x + y</code></pre><p>The example above would result in two consecutive code-chunks.</p><div class="admonition tip"><div class="admonition-title">Tip</div><div class="admonition-text"><p>The rest of the line, after <code>#-</code>, is discarded, so it is possible to use e.g. <code>#-------------</code> as a chunk splitter, which may make the source code more readable.</p></div></div><h2><a class="nav-anchor" id="Document-generation-1" href="#Document-generation-1"><strong>3.3.</strong> Document generation</a></h2><p>After the parsing it is time to generate the output. What is done in this step is very different depending on the output target, and it is describe in more detail in the Output format sections: <a href="outputformats.html#Markdown-Output-1">Markdown Output</a>, <a href="outputformats.html#Notebook-Output-1">Notebook Output</a> and <a href="outputformats.html#Script-Output-1">Script Output</a>. Using the default settings, the following is happening:</p><ul><li><p>Markdown output: markdown chunks are printed as-is, code chunks are put inside a code fence (defaults to <code>@example</code>-blocks),</p></li><li><p>Notebook output: markdown chunks are printed in markdown cells, code chunks are put in code cells,</p></li><li><p>Script output: markdown chunks are discarded, code chunks are printed as-is.</p></li></ul><h2><a class="nav-anchor" id="Post-processing-1" href="#Post-processing-1"><strong>3.4.</strong> Post-processing</a></h2><p>When the document is generated the user, again, has the option to hook-into the generation with a custom post-processing function. The reason is that one might want to change things that are only visible in the rendered document. See <a href="customprocessing.html#Custom-pre-and-post-processing-1">Custom pre- and post-processing</a>.</p><h2><a class="nav-anchor" id="Writing-to-file-1" href="#Writing-to-file-1"><strong>3.5.</strong> Writing to file</a></h2><p>The last step of the generation is writing to file. The result is written to <code>$(outputdir)/$(name)(.md|.ipynb|.jl)</code> where <code>outputdir</code> is the output directory supplied by the user (for example <code>docs/generated</code>), and <code>name</code> is a user supplied filename. It is recommended to add the output directory to <code>.gitignore</code> since the idea is that the generated documents will be generated as part of the build process rather than beeing files in the repo.</p><footer><hr/><a class="previous" href="fileformat.html"><span class="direction">Previous</span><span class="title"><strong>2.</strong> File Format</span></a><a class="next" href="outputformats.html"><span class="direction">Next</span><span class="title"><strong>4.</strong> Output Formats</span></a></footer></article></body></html>
|
|
|
|
