5. Custom pre- and post-processing
Since all packages are different, and may have different demands on how to create a nice example for the documentation it is important that the package maintainer does not feel limited by the by default provided syntax that this package offers. While you can generally come a long way by utilizing line filtering there might be situations where you need to manually hook into the generation and change things. In Literate this is done by letting the user supply custom pre- and post-processing functions that may do transformation of the content.
All of the generators (Literate.markdown, Literate.notebook and Literate.script) accepts preprocess and postprocess keyword arguments. The default "transformation" is the identity function. The input to the transformation functions is a String, and the output should be the transformed String.
preprocess is sent the raw input that is read from the source file (modulo the default line ending transformation). postprocess is given different things depending on the output: For markdown and script output postprocess is given the content String just before writing it to the output file, but for notebook output postprocess is given the dictionary representing the notebook, since, in general, this is more useful.
As an example, lets say we want to splice the date of generation into the output. We could of course update our source file before generating the docs, but we could instead use a preprocess function that splices the date into the source for us. Consider the following source file:
#' # Example
-#' This example was generated DATEOFTODAY
+5. Custom pre- and post-processing · Literate.jl
5. Custom pre- and post-processing
Since all packages are different, and may have different demands on how to create a nice example for the documentation it is important that the package maintainer does not feel limited by the by default provided syntax that this package offers. While you can generally come a long way by utilizing line filtering there might be situations where you need to manually hook into the generation and change things. In Literate this is done by letting the user supply custom pre- and post-processing functions that may do transformation of the content.
All of the generators (Literate.markdown, Literate.notebook and Literate.script) accepts preprocess and postprocess keyword arguments. The default "transformation" is the identity function. The input to the transformation functions is a String, and the output should be the transformed String.
preprocess is sent the raw input that is read from the source file (modulo the default line ending transformation). postprocess is given different things depending on the output: For markdown and script output postprocess is given the content String just before writing it to the output file, but for notebook output postprocess is given the dictionary representing the notebook, since, in general, this is more useful.
As an example, lets say we want to splice the date of generation into the output. We could of course update our source file before generating the docs, but we could instead use a preprocess function that splices the date into the source for us. Consider the following source file:
# # Example
+# This example was generated DATEOFTODAY
x = 1 // 3
where DATEOFTODAY is a placeholder, to make it easier for our preprocess function to find the location. Now, lets define the preprocess function, for example
function update_date(content)
content = replace(content, "DATEOFTODAY" => Date(now()))
diff --git a/latest/fileformat.html b/latest/fileformat.html
index 2ea4848..a194250 100644
--- a/latest/fileformat.html
+++ b/latest/fileformat.html
@@ -1,17 +1,17 @@
-2. File Format · Literate.jl
2. File Format
The source file format for Literate is a regular, commented, julia (.jl) scripts. The idea is that the scripts also serve as documentation on their own and it is also simple to include them in the test-suite, with e.g. include, to make sure the examples stay up do date with other changes in your package.
2.1. Syntax
The basic syntax is simple:
lines starting with #' is treated as markdown,
all other lines are treated as julia code.
The reason for using #' instead of # is that we want to be able to use # as comments, just as in a regular script. Lets look at a simple example:
#' # Rational numbers
-#'
-#' In julia rational numbers can be constructed with the `//` operator.
-#' Lets define two rational numbers, `x` and `y`:
+2. File Format · Literate.jl
2. File Format
The source file format for Literate is a regular, commented, julia (.jl) scripts. The idea is that the scripts also serve as documentation on their own and it is also simple to include them in the test-suite, with e.g. include, to make sure the examples stay up do date with other changes in your package.
2.1. Syntax
The basic syntax is simple:
lines starting with # are treated as markdown,
all other lines are treated as julia code.
NoteIf you want regular julia comments in the source file use ## instead of #.
Lets look at a simple example:
# # Rational numbers
+#
+# In julia rational numbers can be constructed with the `//` operator.
+# Lets define two rational numbers, `x` and `y`:
x = 1//3
y = 2//5
-#' When adding `x` and `y` together we obtain a new rational number:
+# When adding `x` and `y` together we obtain a new rational number:
-z = x + y
In the lines #' we can use regular markdown syntax, for example the # used for the heading and the backticks for formatting code. The other lines are regular julia code. We note a couple of things:
The script is valid julia, which means that we can include it and the example will run (for example in the test/runtests.jl script, to include the example in the test suite).
The script is "self-explanatory", i.e. the markdown lines works as comments and thus serve as good documentation on its own.
For simple use this is all you need to know. The following additional special syntax can also be used:
#md, #nb, #jl, #src: tags to filter lines, see Filtering Lines,
#-: tag to manually control chunk-splits, see Custom control over chunk splits.
There is also some default convenience replacements that will always be performed, see Default Replacements.
2.2. Filtering Lines
It is often useful to filter out lines in the source depending on the output format. For this purpose there are a number of "tokens" that can be used to mark the purpose of certain lines:
#md: line exclusive to markdown output,
#nb: line exclusive to notebook output,
#jl: line exclusive to script output,
#src: line exclusive to the source code and thus filtered out unconditionally.
Lines starting with one of these tokens are filtered out in the preprocessing step.
Suppose, for example, that we want to include a docstring within a @docs block using Documenter. Obviously we don't want to include this in the notebook, since @docs is Documenter syntax that the notebook will not understand. This is a case where we can prepend #md to those lines:
#md #' ```@docs
-#md #' Literate.markdown
-#md #' Literate.notebook
-#md #' Literate.markdown
-#md #' ```
The lines in the example above would be filtered out in the preprocessing step, unless we are generating a markdown file. When generating a markdown file we would simple remove the leading #md from the lines. Beware that the space after the tag is also removed.
The #src token can also be placed at the end of a line. This is to make it possible to have code lines exclusive to the source code, and not just comment lines. For example, if the source file is included in the test suite we might want to add a @test at the end without this showing up in the outputs:
using Test #src
+z = x + y
In the lines starting with # we can use regular markdown syntax, for example the # used for the heading and the backticks for formatting code. The other lines are regular julia code. We note a couple of things:
The script is valid julia, which means that we can include it and the example will run (for example in the test/runtests.jl script, to include the example in the test suite).
The script is "self-explanatory", i.e. the markdown lines works as comments and thus serve as good documentation on its own.
For simple use this is all you need to know. The following additional special syntax can also be used:
#md, #nb, #jl, #src: tags to filter lines, see Filtering Lines,
#-: tag to manually control chunk-splits, see Custom control over chunk splits.
There is also some default convenience replacements that will always be performed, see Default Replacements.
2.2. Filtering Lines
It is often useful to filter out lines in the source depending on the output format. For this purpose there are a number of "tokens" that can be used to mark the purpose of certain lines:
#md: line exclusive to markdown output,
#nb: line exclusive to notebook output,
#jl: line exclusive to script output,
#src: line exclusive to the source code and thus filtered out unconditionally.
Lines starting with one of these tokens are filtered out in the preprocessing step.
Suppose, for example, that we want to include a docstring within a @docs block using Documenter. Obviously we don't want to include this in the notebook, since @docs is Documenter syntax that the notebook will not understand. This is a case where we can prepend #md to those lines:
#md # ```@docs
+#md # Literate.markdown
+#md # Literate.notebook
+#md # Literate.markdown
+#md # ```
The lines in the example above would be filtered out in the preprocessing step, unless we are generating a markdown file. When generating a markdown file we would simple remove the leading #md from the lines. Beware that the space after the tag is also removed.
The #src token can also be placed at the end of a line. This is to make it possible to have code lines exclusive to the source code, and not just comment lines. For example, if the source file is included in the test suite we might want to add a @test at the end without this showing up in the outputs:
using Test #src
@test result == expected_result #src
2.3. Default Replacements
The following convenience "macros" are always expanded:
@__NAME__
expands to the name keyword argument to Literate.markdown, Literate.notebook and Literate.script (defaults to the filename of the input file).
@__REPO__ROOT_URL__
expands to https://github.com/$(ENV["TRAVIS_REPO_SLUG"])/blob/master/ and is a convenient way to use when you want to link to files outside the doc-build directory. For example @__REPO__ROOT_URL__src/Literate.jl would link to the source of the Literate module.
@__NBVIEWER_ROOT_URL__
expands to https://nbviewer.jupyter.org/github/$(ENV["TRAVIS_REPO_SLUG"])/blob/gh-pages/$(folder)/ where folder is the folder that Documenter.deploydocs deploys too. This can be used if you want a link that opens the generated notebook in http://nbviewer.jupyter.org/.
diff --git a/latest/generated/example.html b/latest/generated/example.html
index 7103e2d..b6b5d75 100644
--- a/latest/generated/example.html
+++ b/latest/generated/example.html
@@ -1,5 +1,5 @@
-7. Example · Literate.jl
7. Example
This is an example generated with Literate based on this source file: example.jl. You are seeing the html-output which Documenter have generated based on a markdown file generated with Literate. The corresponding notebook can be found here: example.ipynb, and the plain script output can be found here: example.jl.
It is recommended to have the source file available when reading this, to better understand how the syntax in the source file corresponds to the output you are seeing.
Basic syntax
The basic syntax for Literate is simple, lines starting with #' is interpreted as markdown, and all the other lines are interpreted as code. Here is some code:
x = 1//3
+7. Example · Literate.jl
7. Example
This is an example generated with Literate based on this source file: example.jl. You are seeing the html-output which Documenter have generated based on a markdown file generated with Literate. The corresponding notebook can be found here: example.ipynb, and the plain script output can be found here: example.jl.
It is recommended to have the source file available when reading this, to better understand how the syntax in the source file corresponds to the output you are seeing.
Basic syntax
The basic syntax for Literate is simple, lines starting with # is interpreted as markdown, and all the other lines are interpreted as code. Here is some code:
x = 1//3
y = 2//5
2//5
In markdown sections we can use markdown syntax. For example, we can write text in italic font, text in bold font and use links.
It is possible to filter out lines depending on the output using the #md, #nb, #jl and #src tags (see Filtering Lines):
This line starts with #md and is thus only visible in the markdown output.
The source file is parsed in chunks of markdown and code. Starting a line with #- manually inserts a chunk break. For example, if we want to display the output of the following operations we may insert #- in between. These two code blocks will now end up in different @example-blocks in the markdown output, and two different notebook cells in the notebook output.
x + y
11//15
x * y
2//15
Output Capturing
Code chunks are by default placed in Documenter @example blocks in the generated markdown. This means that the output will be captured in a block when Documenter is building the docs. In notebooks the output is captured in output cells, if the execute keyword argument is set to true. Output to stdout/stderr is also captured.
function foo()
println("This string is printed to stdout.")
return [1, 2, 3, 4]
diff --git a/latest/generated/example.ipynb b/latest/generated/example.ipynb
index 37b7a0d..4a7e22b 100644
--- a/latest/generated/example.ipynb
+++ b/latest/generated/example.ipynb
@@ -30,7 +30,7 @@
"cell_type": "markdown",
"source": [
"### Basic syntax\n",
- "The basic syntax for Literate is simple, lines starting with `#'` is interpreted\n",
+ "The basic syntax for Literate is simple, lines starting with `# ` is interpreted\n",
"as markdown, and all the other lines are interpreted as code. Here is some code:"
],
"metadata": {}
diff --git a/latest/outputformats.html b/latest/outputformats.html
index 00c87c4..1987e5b 100644
--- a/latest/outputformats.html
+++ b/latest/outputformats.html
@@ -1,14 +1,14 @@
-4. Output Formats · Literate.jl
4. Output Formats
When the source is parsed, and have been processed it is time to render the output. We will consider the following source snippet:
#' # Rational numbers
-#'
-#' In julia rational numbers can be constructed with the `//` operator.
-#' Lets define two rational numbers, `x` and `y`:
+4. Output Formats · Literate.jl
4. Output Formats
When the source is parsed, and have been processed it is time to render the output. We will consider the following source snippet:
# # Rational numbers
+#
+# In julia rational numbers can be constructed with the `//` operator.
+# Lets define two rational numbers, `x` and `y`:
x = 1//3
#-
y = 2//5
-#' When adding `x` and `y` together we obtain a new rational number:
+# When adding `x` and `y` together we obtain a new rational number:
z = x + y
and see how this is rendered in each of the output formats.
4.1. Markdown Output
The (default) markdown output of the source snippet above is as follows
```@meta
EditURL = "https://github.com/fredrikekre/Literate.jl/blob/master/docs/src/outputformats.jl"
@@ -31,8 +31,8 @@ When adding `x` and `y` together we obtain a new rational number:
```@example name
z = x + y
-```
We note that lines starting with #' is printed as regular markdown, and the code lines have been wrapped in @example blocks. We also note that an @meta block have been added, that sets the EditURL variable. This is used by Documenter to redirect the "Edit on GitHub" link for the page, see Interaction with Documenter.
Some of the output rendering can be controlled with keyword arguments to Literate.markdown:
Literate.markdown — Function.Literate.markdown(inputfile, outputdir; kwargs...)
Generate a markdown file from inputfile and write the result to the directoryoutputdir.
Keyword arguments:
name: name of the output file, excluding .md. name is also used to name all the @example blocks, and to replace @__NAME__. Defaults to the filename of inputfile.
preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.
documenter: boolean that tells if the output is intended to use with Documenter.jl. Defaults to true. See the the manual section on Interaction with Documenter.
codefence: A Pair of opening and closing code fence. Defaults to
"```@example $(name)" => "```"
if documenter = true and
"```julia" => "```"
if documenter = false.
source4.2. Notebook Output
The (default) notebook output of the source snippet can be seen here: notebook.ipynb.
We note that lines starting with #' is put in markdown cells, and the code lines have been put in code cells. By default the notebook is also executed and output cells populated. The current working directory is set to the specified output directory the notebook is executed. Some of the output rendering can be controlled with keyword arguments to Literate.notebook:
Literate.notebook — Function.Literate.notebook(inputfile, outputdir; kwargs...)
Generate a notebook from inputfile and write the result to outputdir.
Keyword arguments:
name: name of the output file, excluding .ipynb. name is also used to replace @__NAME__. Defaults to the filename of inputfile.
preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.
execute: a boolean deciding if the generated notebook should also be executed or not. Defaults to true. The current working directory is set to outputdir when executing the notebook.
documenter: boolean that says if the source contains Documenter.jl specific things to filter out during notebook generation. Defaults to true. See the the manual section on Interaction with Documenter.
source4.3. Script Output
The (default) script output of the source snippet above is as follows
x = 1//3
+```
We note that lines starting with # are printed as regular markdown, and the code lines have been wrapped in @example blocks. We also note that an @meta block have been added, that sets the EditURL variable. This is used by Documenter to redirect the "Edit on GitHub" link for the page, see Interaction with Documenter.
Some of the output rendering can be controlled with keyword arguments to Literate.markdown:
Literate.markdown — Function.Literate.markdown(inputfile, outputdir; kwargs...)
Generate a markdown file from inputfile and write the result to the directoryoutputdir.
Keyword arguments:
name: name of the output file, excluding .md. name is also used to name all the @example blocks, and to replace @__NAME__. Defaults to the filename of inputfile.
preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.
documenter: boolean that tells if the output is intended to use with Documenter.jl. Defaults to true. See the the manual section on Interaction with Documenter.
codefence: A Pair of opening and closing code fence. Defaults to
"```@example $(name)" => "```"
if documenter = true and
"```julia" => "```"
if documenter = false.
source4.2. Notebook Output
The (default) notebook output of the source snippet can be seen here: notebook.ipynb.
We note that lines starting with # are placed in markdown cells, and the code lines have been placed in code cells. By default the notebook is also executed and output cells populated. The current working directory is set to the specified output directory the notebook is executed. Some of the output rendering can be controlled with keyword arguments to Literate.notebook:
Literate.notebook — Function.Literate.notebook(inputfile, outputdir; kwargs...)
Generate a notebook from inputfile and write the result to outputdir.
Keyword arguments:
name: name of the output file, excluding .ipynb. name is also used to replace @__NAME__. Defaults to the filename of inputfile.
preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.
execute: a boolean deciding if the generated notebook should also be executed or not. Defaults to true. The current working directory is set to outputdir when executing the notebook.
documenter: boolean that says if the source contains Documenter.jl specific things to filter out during notebook generation. Defaults to true. See the the manual section on Interaction with Documenter.
source4.3. Script Output
The (default) script output of the source snippet above is as follows
x = 1//3
y = 2//5
-z = x + y
We note that lines starting with #' are removed and only the code lines have been kept. Some of the output rendering can be controlled with keyword arguments to Literate.script:
Literate.script — Function.Literate.script(inputfile, outputdir; kwargs...)
Generate a plain script file from inputfile and write the result to outputdir.
Keyword arguments:
name: name of the output file, excluding .jl. name is also used to replace @__NAME__. Defaults to the filename of inputfile.
preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.
documenter: boolean that says if the source contains Documenter.jl specific things to filter out during script generation. Defaults to true. See the the manual section on Interaction with Documenter.
keep_comments: boolean that, if set to true, keeps markdown lines (#') as comments in the output script. Defaults to false.
source
+z = x + y
We note that lines starting with # are removed and only the code lines have been kept. Some of the output rendering can be controlled with keyword arguments to Literate.script:
Literate.script — Function.Literate.script(inputfile, outputdir; kwargs...)
Generate a plain script file from inputfile and write the result to outputdir.
Keyword arguments:
name: name of the output file, excluding .jl. name is also used to replace @__NAME__. Defaults to the filename of inputfile.
preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.
documenter: boolean that says if the source contains Documenter.jl specific things to filter out during script generation. Defaults to true. See the the manual section on Interaction with Documenter.
keep_comments: boolean that, if set to true, keeps markdown lines as comments in the output script. Defaults to false.
source
diff --git a/latest/outputformats.jl b/latest/outputformats.jl
index 4fbfd71..9dbdad1 100644
--- a/latest/outputformats.jl
+++ b/latest/outputformats.jl
@@ -1,12 +1,12 @@
-#' # Rational numbers
-#'
-#' In julia rational numbers can be constructed with the `//` operator.
-#' Lets define two rational numbers, `x` and `y`:
+# # Rational numbers
+#
+# In julia rational numbers can be constructed with the `//` operator.
+# Lets define two rational numbers, `x` and `y`:
x = 1//3
#-
y = 2//5
-#' When adding `x` and `y` together we obtain a new rational number:
+# When adding `x` and `y` together we obtain a new rational number:
z = x + y
diff --git a/latest/pipeline.html b/latest/pipeline.html
index 6d11149..071a080 100644
--- a/latest/pipeline.html
+++ b/latest/pipeline.html
@@ -1,29 +1,29 @@
-3. Processing pipeline · Literate.jl
3. Processing pipeline
The generation of output follows the same pipeline for all output formats:
3.1. Pre-processing
The first step is pre-processing of the input file. The file is read to a String. The first processing step is to apply the user specified pre-processing function, see Custom pre- and post-processing.
The next step is to perform all of the built-in default replacements. CRLF style line endings ("\r\n") are replaced with LF line endings ("\n") to simplify internal processing. Next, line filtering is performed, see Filtering Lines, meaning that lines starting with #md, #nb or #jl are handled (either just the token itself is removed, or the full line, depending on the output target). The last pre-processing step is to expand the convenience "macros" described in Default Replacements is expanded.
3.2. Parsing
After the preprocessing the file is parsed. The first step is to categorize each line and mark them as either markdown or code according to the rules described in the Syntax section. Lets consider the example from the previous section with each line categorized:
#' # Rational numbers <- markdown
-#' <- markdown
-#' In julia rational numbers can be constructed with the `//` operator. <- markdown
-#' Lets define two rational numbers, `x` and `y`: <- markdown
- <- code
-x = 1 // 3 <- code
-y = 2 // 5 <- code
- <- code
-#' When adding `x` and `y` together we obtain a new rational number: <- markdown
- <- code
-z = x + y <- code
In the next step the lines are grouped into "chunks" of markdown and code. This is done by simply collecting adjacent lines of the same "type" into chunks:
#' # Rational numbers ┐
-#' │
-#' In julia rational numbers can be constructed with the `//` operator. │ markdown
-#' Lets define two rational numbers, `x` and `y`: ┘
- ┐
-x = 1 // 3 │
-y = 2 // 5 │ code
- ┘
-#' When adding `x` and `y` together we obtain a new rational number: ] markdown
- ┐
-z = x + y ┘ code
In the last parsing step all empty leading and trailing lines for each chunk are removed, but empty lines within the same block are kept. The leading #' tokens are also removed from the markdown chunks. Finally we would end up with the following 4 chunks:
Chunks #1:
# Rational numbers
+3. Processing pipeline · Literate.jl
3. Processing pipeline
The generation of output follows the same pipeline for all output formats:
3.1. Pre-processing
The first step is pre-processing of the input file. The file is read to a String. The first processing step is to apply the user specified pre-processing function, see Custom pre- and post-processing.
The next step is to perform all of the built-in default replacements. CRLF style line endings ("\r\n") are replaced with LF line endings ("\n") to simplify internal processing. Next, line filtering is performed, see Filtering Lines, meaning that lines starting with #md, #nb or #jl are handled (either just the token itself is removed, or the full line, depending on the output target). The last pre-processing step is to expand the convenience "macros" described in Default Replacements is expanded.
3.2. Parsing
After the preprocessing the file is parsed. The first step is to categorize each line and mark them as either markdown or code according to the rules described in the Syntax section. Lets consider the example from the previous section with each line categorized:
# # Rational numbers <- markdown
+# <- markdown
+# In julia rational numbers can be constructed with the `//` operator. <- markdown
+# Lets define two rational numbers, `x` and `y`: <- markdown
+ <- code
+x = 1 // 3 <- code
+y = 2 // 5 <- code
+ <- code
+# When adding `x` and `y` together we obtain a new rational number: <- markdown
+ <- code
+z = x + y <- code
In the next step the lines are grouped into "chunks" of markdown and code. This is done by simply collecting adjacent lines of the same "type" into chunks:
# # Rational numbers ┐
+# │
+# In julia rational numbers can be constructed with the `//` operator. │ markdown
+# Lets define two rational numbers, `x` and `y`: ┘
+ ┐
+x = 1 // 3 │
+y = 2 // 5 │ code
+ ┘
+# When adding `x` and `y` together we obtain a new rational number: ] markdown
+ ┐
+z = x + y ┘ code
In the last parsing step all empty leading and trailing lines for each chunk are removed, but empty lines within the same block are kept. The leading # tokens are also removed from the markdown chunks. Finally we would end up with the following 4 chunks:
Chunks #1:
# Rational numbers
In julia rational numbers can be constructed with the `//` operator.
Lets define two rational numbers, `x` and `y`:
Chunk #2:
x = 1 // 3
y = 2 // 5
Chunk #3:
When adding `x` and `y` together we obtain a new rational number:
Chunk #4:
z = x + y
It is then up to the Document generation step to decide how these chunks should be treated.
Custom control over chunk splits
Sometimes it is convenient to be able to manually control how the chunks are split. For example, if you want to split a block of code into two, such that they end up in two different @example blocks or notebook cells. The #- token can be used for this purpose. All lines starting with #- are used as "chunk-splitters":
x = 1 // 3
y = 2 // 5
#-
-z = x + y
The example above would result in two consecutive code-chunks.
TipThe rest of the line, after #-, is discarded, so it is possible to use e.g. #------------- as a chunk splitter, which may make the source code more readable.
3.3. Document generation
After the parsing it is time to generate the output. What is done in this step is very different depending on the output target, and it is describe in more detail in the Output format sections: Markdown Output, Notebook Output and Script Output. In short, the following is happening:
Markdown output: markdown chunks are printed as-is, code chunks are put inside a code fence (defaults to @example-blocks),
Notebook output: markdown chunks are printed in markdown cells, code chunks are put in code cells,
Script output: markdown chunks are discarded, code chunks are printed as-is.
3.4. Post-processing
When the document is generated the user, again, has the option to hook-into the generation with a custom post-processing function. The reason is that one might want to change things that are only visible in the rendered document. See Custom pre- and post-processing.
3.5. Writing to file
The last step of the generation is writing to file. The result is written to $(outputdir)/$(name)(.md|.ipynb|.jl) where outputdir is the output directory supplied by the user (for example docs/generated), and name is a user supplied filename. It is recommended to add the output directory to .gitignore since the idea is that the generated documents will be generated as part of the build process rather than beeing files in the repo.
+z = x + y
The example above would result in two consecutive code-chunks.
TipThe rest of the line, after #-, is discarded, so it is possible to use e.g. #------------- as a chunk splitter, which may make the source code more readable.
3.3. Document generation
After the parsing it is time to generate the output. What is done in this step is very different depending on the output target, and it is describe in more detail in the Output format sections: Markdown Output, Notebook Output and Script Output. Using the default settings, the following is happening:
Markdown output: markdown chunks are printed as-is, code chunks are put inside a code fence (defaults to @example-blocks),
Notebook output: markdown chunks are printed in markdown cells, code chunks are put in code cells,
Script output: markdown chunks are discarded, code chunks are printed as-is.
3.4. Post-processing
When the document is generated the user, again, has the option to hook-into the generation with a custom post-processing function. The reason is that one might want to change things that are only visible in the rendered document. See Custom pre- and post-processing.
3.5. Writing to file
The last step of the generation is writing to file. The result is written to $(outputdir)/$(name)(.md|.ipynb|.jl) where outputdir is the output directory supplied by the user (for example docs/generated), and name is a user supplied filename. It is recommended to add the output directory to .gitignore since the idea is that the generated documents will be generated as part of the build process rather than beeing files in the repo.
diff --git a/latest/search_index.js b/latest/search_index.js
index be397e6..21aae8c 100644
--- a/latest/search_index.js
+++ b/latest/search_index.js
@@ -53,7 +53,7 @@ var documenterSearchIndex = {"docs": [
"page": "2. File Format",
"title": "2.1. Syntax",
"category": "section",
- "text": "The basic syntax is simple:lines starting with #\' is treated as markdown,\nall other lines are treated as julia code.The reason for using #\' instead of # is that we want to be able to use # as comments, just as in a regular script. Lets look at a simple example:#\' # Rational numbers\n#\'\n#\' In julia rational numbers can be constructed with the `//` operator.\n#\' Lets define two rational numbers, `x` and `y`:\n\nx = 1//3\ny = 2//5\n\n#\' When adding `x` and `y` together we obtain a new rational number:\n\nz = x + yIn the lines #\' we can use regular markdown syntax, for example the # used for the heading and the backticks for formatting code. The other lines are regular julia code. We note a couple of things:The script is valid julia, which means that we can include it and the example will run (for example in the test/runtests.jl script, to include the example in the test suite).\nThe script is \"self-explanatory\", i.e. the markdown lines works as comments and thus serve as good documentation on its own.For simple use this is all you need to know. The following additional special syntax can also be used:#md, #nb, #jl, #src: tags to filter lines, see Filtering Lines,\n#-: tag to manually control chunk-splits, see Custom control over chunk splits.There is also some default convenience replacements that will always be performed, see Default Replacements."
+ "text": "The basic syntax is simple:lines starting with # are treated as markdown,\nall other lines are treated as julia code.note: Note\nIf you want regular julia comments in the source file use ## instead of #.Lets look at a simple example:# # Rational numbers\n#\n# In julia rational numbers can be constructed with the `//` operator.\n# Lets define two rational numbers, `x` and `y`:\n\nx = 1//3\ny = 2//5\n\n# When adding `x` and `y` together we obtain a new rational number:\n\nz = x + yIn the lines starting with # we can use regular markdown syntax, for example the # used for the heading and the backticks for formatting code. The other lines are regular julia code. We note a couple of things:The script is valid julia, which means that we can include it and the example will run (for example in the test/runtests.jl script, to include the example in the test suite).\nThe script is \"self-explanatory\", i.e. the markdown lines works as comments and thus serve as good documentation on its own.For simple use this is all you need to know. The following additional special syntax can also be used:#md, #nb, #jl, #src: tags to filter lines, see Filtering Lines,\n#-: tag to manually control chunk-splits, see Custom control over chunk splits.There is also some default convenience replacements that will always be performed, see Default Replacements."
},
{
@@ -61,7 +61,7 @@ var documenterSearchIndex = {"docs": [
"page": "2. File Format",
"title": "2.2. Filtering Lines",
"category": "section",
- "text": "It is often useful to filter out lines in the source depending on the output format. For this purpose there are a number of \"tokens\" that can be used to mark the purpose of certain lines:#md: line exclusive to markdown output,\n#nb: line exclusive to notebook output,\n#jl: line exclusive to script output,\n#src: line exclusive to the source code and thus filtered out unconditionally.Lines starting with one of these tokens are filtered out in the preprocessing step.Suppose, for example, that we want to include a docstring within a @docs block using Documenter. Obviously we don\'t want to include this in the notebook, since @docs is Documenter syntax that the notebook will not understand. This is a case where we can prepend #md to those lines:#md #\' ```@docs\n#md #\' Literate.markdown\n#md #\' Literate.notebook\n#md #\' Literate.markdown\n#md #\' ```The lines in the example above would be filtered out in the preprocessing step, unless we are generating a markdown file. When generating a markdown file we would simple remove the leading #md from the lines. Beware that the space after the tag is also removed.The #src token can also be placed at the end of a line. This is to make it possible to have code lines exclusive to the source code, and not just comment lines. For example, if the source file is included in the test suite we might want to add a @test at the end without this showing up in the outputs:using Test #src\n@test result == expected_result #src"
+ "text": "It is often useful to filter out lines in the source depending on the output format. For this purpose there are a number of \"tokens\" that can be used to mark the purpose of certain lines:#md: line exclusive to markdown output,\n#nb: line exclusive to notebook output,\n#jl: line exclusive to script output,\n#src: line exclusive to the source code and thus filtered out unconditionally.Lines starting with one of these tokens are filtered out in the preprocessing step.Suppose, for example, that we want to include a docstring within a @docs block using Documenter. Obviously we don\'t want to include this in the notebook, since @docs is Documenter syntax that the notebook will not understand. This is a case where we can prepend #md to those lines:#md # ```@docs\n#md # Literate.markdown\n#md # Literate.notebook\n#md # Literate.markdown\n#md # ```The lines in the example above would be filtered out in the preprocessing step, unless we are generating a markdown file. When generating a markdown file we would simple remove the leading #md from the lines. Beware that the space after the tag is also removed.The #src token can also be placed at the end of a line. This is to make it possible to have code lines exclusive to the source code, and not just comment lines. For example, if the source file is included in the test suite we might want to add a @test at the end without this showing up in the outputs:using Test #src\n@test result == expected_result #src"
},
{
@@ -101,7 +101,7 @@ var documenterSearchIndex = {"docs": [
"page": "3. Processing pipeline",
"title": "3.2. Parsing",
"category": "section",
- "text": "After the preprocessing the file is parsed. The first step is to categorize each line and mark them as either markdown or code according to the rules described in the Syntax section. Lets consider the example from the previous section with each line categorized:#\' # Rational numbers <- markdown\n#\' <- markdown\n#\' In julia rational numbers can be constructed with the `//` operator. <- markdown\n#\' Lets define two rational numbers, `x` and `y`: <- markdown\n <- code\nx = 1 // 3 <- code\ny = 2 // 5 <- code\n <- code\n#\' When adding `x` and `y` together we obtain a new rational number: <- markdown\n <- code\nz = x + y <- codeIn the next step the lines are grouped into \"chunks\" of markdown and code. This is done by simply collecting adjacent lines of the same \"type\" into chunks:#\' # Rational numbers ┐\n#\' │\n#\' In julia rational numbers can be constructed with the `//` operator. │ markdown\n#\' Lets define two rational numbers, `x` and `y`: ┘\n ┐\nx = 1 // 3 │\ny = 2 // 5 │ code\n ┘\n#\' When adding `x` and `y` together we obtain a new rational number: ] markdown\n ┐\nz = x + y ┘ codeIn the last parsing step all empty leading and trailing lines for each chunk are removed, but empty lines within the same block are kept. The leading #\' tokens are also removed from the markdown chunks. Finally we would end up with the following 4 chunks:Chunks #1:# Rational numbers\n\nIn julia rational numbers can be constructed with the `//` operator.\nLets define two rational numbers, `x` and `y`:Chunk #2:x = 1 // 3\ny = 2 // 5Chunk #3:When adding `x` and `y` together we obtain a new rational number:Chunk #4:z = x + yIt is then up to the Document generation step to decide how these chunks should be treated."
+ "text": "After the preprocessing the file is parsed. The first step is to categorize each line and mark them as either markdown or code according to the rules described in the Syntax section. Lets consider the example from the previous section with each line categorized:# # Rational numbers <- markdown\n# <- markdown\n# In julia rational numbers can be constructed with the `//` operator. <- markdown\n# Lets define two rational numbers, `x` and `y`: <- markdown\n <- code\nx = 1 // 3 <- code\ny = 2 // 5 <- code\n <- code\n# When adding `x` and `y` together we obtain a new rational number: <- markdown\n <- code\nz = x + y <- codeIn the next step the lines are grouped into \"chunks\" of markdown and code. This is done by simply collecting adjacent lines of the same \"type\" into chunks:# # Rational numbers ┐\n# │\n# In julia rational numbers can be constructed with the `//` operator. │ markdown\n# Lets define two rational numbers, `x` and `y`: ┘\n ┐\nx = 1 // 3 │\ny = 2 // 5 │ code\n ┘\n# When adding `x` and `y` together we obtain a new rational number: ] markdown\n ┐\nz = x + y ┘ codeIn the last parsing step all empty leading and trailing lines for each chunk are removed, but empty lines within the same block are kept. The leading # tokens are also removed from the markdown chunks. Finally we would end up with the following 4 chunks:Chunks #1:# Rational numbers\n\nIn julia rational numbers can be constructed with the `//` operator.\nLets define two rational numbers, `x` and `y`:Chunk #2:x = 1 // 3\ny = 2 // 5Chunk #3:When adding `x` and `y` together we obtain a new rational number:Chunk #4:z = x + yIt is then up to the Document generation step to decide how these chunks should be treated."
},
{
@@ -117,7 +117,7 @@ var documenterSearchIndex = {"docs": [
"page": "3. Processing pipeline",
"title": "3.3. Document generation",
"category": "section",
- "text": "After the parsing it is time to generate the output. What is done in this step is very different depending on the output target, and it is describe in more detail in the Output format sections: Markdown Output, Notebook Output and Script Output. In short, the following is happening:Markdown output: markdown chunks are printed as-is, code chunks are put inside a code fence (defaults to @example-blocks),\nNotebook output: markdown chunks are printed in markdown cells, code chunks are put in code cells,\nScript output: markdown chunks are discarded, code chunks are printed as-is."
+ "text": "After the parsing it is time to generate the output. What is done in this step is very different depending on the output target, and it is describe in more detail in the Output format sections: Markdown Output, Notebook Output and Script Output. Using the default settings, the following is happening:Markdown output: markdown chunks are printed as-is, code chunks are put inside a code fence (defaults to @example-blocks),\nNotebook output: markdown chunks are printed in markdown cells, code chunks are put in code cells,\nScript output: markdown chunks are discarded, code chunks are printed as-is."
},
{
@@ -165,7 +165,7 @@ var documenterSearchIndex = {"docs": [
"page": "4. Output Formats",
"title": "4.1. Markdown Output",
"category": "section",
- "text": "The (default) markdown output of the source snippet above is as followsfile = joinpath(@__DIR__, \"src/generated/name.md\")\nstr = \"````markdown\\n\" * rstrip(read(file, String)) * \"\\n````\"\nrm(file)\nMarkdown.parse(str)We note that lines starting with #\' is printed as regular markdown, and the code lines have been wrapped in @example blocks. We also note that an @meta block have been added, that sets the EditURL variable. This is used by Documenter to redirect the \"Edit on GitHub\" link for the page, see Interaction with Documenter.Some of the output rendering can be controlled with keyword arguments to Literate.markdown:Literate.markdown"
+ "text": "The (default) markdown output of the source snippet above is as followsfile = joinpath(@__DIR__, \"src/generated/name.md\")\nstr = \"````markdown\\n\" * rstrip(read(file, String)) * \"\\n````\"\nrm(file)\nMarkdown.parse(str)We note that lines starting with # are printed as regular markdown, and the code lines have been wrapped in @example blocks. We also note that an @meta block have been added, that sets the EditURL variable. This is used by Documenter to redirect the \"Edit on GitHub\" link for the page, see Interaction with Documenter.Some of the output rendering can be controlled with keyword arguments to Literate.markdown:Literate.markdown"
},
{
@@ -181,7 +181,7 @@ var documenterSearchIndex = {"docs": [
"page": "4. Output Formats",
"title": "4.2. Notebook Output",
"category": "section",
- "text": "The (default) notebook output of the source snippet can be seen here: notebook.ipynb.We note that lines starting with #\' is put in markdown cells, and the code lines have been put in code cells. By default the notebook is also executed and output cells populated. The current working directory is set to the specified output directory the notebook is executed. Some of the output rendering can be controlled with keyword arguments to Literate.notebook:Literate.notebook"
+ "text": "The (default) notebook output of the source snippet can be seen here: notebook.ipynb.We note that lines starting with # are placed in markdown cells, and the code lines have been placed in code cells. By default the notebook is also executed and output cells populated. The current working directory is set to the specified output directory the notebook is executed. Some of the output rendering can be controlled with keyword arguments to Literate.notebook:Literate.notebook"
},
{
@@ -189,7 +189,7 @@ var documenterSearchIndex = {"docs": [
"page": "4. Output Formats",
"title": "Literate.script",
"category": "function",
- "text": "Literate.script(inputfile, outputdir; kwargs...)\n\nGenerate a plain script file from inputfile and write the result to outputdir.\n\nKeyword arguments:\n\nname: name of the output file, excluding .jl. name is also used to replace @__NAME__. Defaults to the filename of inputfile.\npreprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.\ndocumenter: boolean that says if the source contains Documenter.jl specific things to filter out during script generation. Defaults to true. See the the manual section on Interaction with Documenter.\nkeep_comments: boolean that, if set to true, keeps markdown lines (#\') as comments in the output script. Defaults to false.\n\n\n\n"
+ "text": "Literate.script(inputfile, outputdir; kwargs...)\n\nGenerate a plain script file from inputfile and write the result to outputdir.\n\nKeyword arguments:\n\nname: name of the output file, excluding .jl. name is also used to replace @__NAME__. Defaults to the filename of inputfile.\npreprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.\ndocumenter: boolean that says if the source contains Documenter.jl specific things to filter out during script generation. Defaults to true. See the the manual section on Interaction with Documenter.\nkeep_comments: boolean that, if set to true, keeps markdown lines as comments in the output script. Defaults to false.\n\n\n\n"
},
{
@@ -197,7 +197,7 @@ var documenterSearchIndex = {"docs": [
"page": "4. Output Formats",
"title": "4.3. Script Output",
"category": "section",
- "text": "The (default) script output of the source snippet above is as followsfile = joinpath(@__DIR__, \"src/generated/outputformats.jl\")\nstr = \"```julia\\n\" * rstrip(read(file, String)) * \"\\n```\"\nrm(file)\nMarkdown.parse(str)We note that lines starting with #\' are removed and only the code lines have been kept. Some of the output rendering can be controlled with keyword arguments to Literate.script:Literate.script"
+ "text": "The (default) script output of the source snippet above is as followsfile = joinpath(@__DIR__, \"src/generated/outputformats.jl\")\nstr = \"```julia\\n\" * rstrip(read(file, String)) * \"\\n```\"\nrm(file)\nMarkdown.parse(str)We note that lines starting with # are removed and only the code lines have been kept. Some of the output rendering can be controlled with keyword arguments to Literate.script:Literate.script"
},
{
@@ -213,7 +213,7 @@ var documenterSearchIndex = {"docs": [
"page": "5. Custom pre- and post-processing",
"title": "5. Custom pre- and post-processing",
"category": "section",
- "text": "Since all packages are different, and may have different demands on how to create a nice example for the documentation it is important that the package maintainer does not feel limited by the by default provided syntax that this package offers. While you can generally come a long way by utilizing line filtering there might be situations where you need to manually hook into the generation and change things. In Literate this is done by letting the user supply custom pre- and post-processing functions that may do transformation of the content.All of the generators (Literate.markdown, Literate.notebook and Literate.script) accepts preprocess and postprocess keyword arguments. The default \"transformation\" is the identity function. The input to the transformation functions is a String, and the output should be the transformed String.preprocess is sent the raw input that is read from the source file (modulo the default line ending transformation). postprocess is given different things depending on the output: For markdown and script output postprocess is given the content String just before writing it to the output file, but for notebook output postprocess is given the dictionary representing the notebook, since, in general, this is more useful.As an example, lets say we want to splice the date of generation into the output. We could of course update our source file before generating the docs, but we could instead use a preprocess function that splices the date into the source for us. Consider the following source file:#\' # Example\n#\' This example was generated DATEOFTODAY\n\nx = 1 // 3where DATEOFTODAY is a placeholder, to make it easier for our preprocess function to find the location. Now, lets define the preprocess function, for examplefunction update_date(content)\n content = replace(content, \"DATEOFTODAY\" => Date(now()))\n return content\nendwhich would replace every occurrence of \"DATEOFTODAY\" with the current date. We would now simply give this function to the generator, for example:Literate.markdown(\"input.jl\", \"outputdir\"; preprocess = update_date)"
+ "text": "Since all packages are different, and may have different demands on how to create a nice example for the documentation it is important that the package maintainer does not feel limited by the by default provided syntax that this package offers. While you can generally come a long way by utilizing line filtering there might be situations where you need to manually hook into the generation and change things. In Literate this is done by letting the user supply custom pre- and post-processing functions that may do transformation of the content.All of the generators (Literate.markdown, Literate.notebook and Literate.script) accepts preprocess and postprocess keyword arguments. The default \"transformation\" is the identity function. The input to the transformation functions is a String, and the output should be the transformed String.preprocess is sent the raw input that is read from the source file (modulo the default line ending transformation). postprocess is given different things depending on the output: For markdown and script output postprocess is given the content String just before writing it to the output file, but for notebook output postprocess is given the dictionary representing the notebook, since, in general, this is more useful.As an example, lets say we want to splice the date of generation into the output. We could of course update our source file before generating the docs, but we could instead use a preprocess function that splices the date into the source for us. Consider the following source file:# # Example\n# This example was generated DATEOFTODAY\n\nx = 1 // 3where DATEOFTODAY is a placeholder, to make it easier for our preprocess function to find the location. Now, lets define the preprocess function, for examplefunction update_date(content)\n content = replace(content, \"DATEOFTODAY\" => Date(now()))\n return content\nendwhich would replace every occurrence of \"DATEOFTODAY\" with the current date. We would now simply give this function to the generator, for example:Literate.markdown(\"input.jl\", \"outputdir\"; preprocess = update_date)"
},
{
@@ -277,7 +277,7 @@ var documenterSearchIndex = {"docs": [
"page": "7. Example",
"title": "Basic syntax",
"category": "section",
- "text": "The basic syntax for Literate is simple, lines starting with #\' is interpreted as markdown, and all the other lines are interpreted as code. Here is some code:x = 1//3\ny = 2//5In markdown sections we can use markdown syntax. For example, we can write text in italic font, text in bold font and use links.It is possible to filter out lines depending on the output using the #md, #nb, #jl and #src tags (see Filtering Lines):This line starts with #md and is thus only visible in the markdown output.The source file is parsed in chunks of markdown and code. Starting a line with #- manually inserts a chunk break. For example, if we want to display the output of the following operations we may insert #- in between. These two code blocks will now end up in different @example-blocks in the markdown output, and two different notebook cells in the notebook output.x + yx * y"
+ "text": "The basic syntax for Literate is simple, lines starting with # is interpreted as markdown, and all the other lines are interpreted as code. Here is some code:x = 1//3\ny = 2//5In markdown sections we can use markdown syntax. For example, we can write text in italic font, text in bold font and use links.It is possible to filter out lines depending on the output using the #md, #nb, #jl and #src tags (see Filtering Lines):This line starts with #md and is thus only visible in the markdown output.The source file is parsed in chunks of markdown and code. Starting a line with #- manually inserts a chunk break. For example, if we want to display the output of the following operations we may insert #- in between. These two code blocks will now end up in different @example-blocks in the markdown output, and two different notebook cells in the notebook output.x + yx * y"
},
{