5. Custom pre- and post-processing
Since all packages are different, and may have different demands on how to create a nice example for the documentation it is important that the package maintainer does not feel limited by the by default provided syntax that this package offers. While you can generally come a long way by utilizing line filtering there might be situations where you need to manually hook into the generation and change things. In Literate this is done by letting the user supply custom pre- and post-processing functions that may do transformation of the content.
All of the generators (Literate.markdown, Literate.notebook and Literate.script) accepts preprocess and postprocess keyword arguments. The default "transformation" is the identity function. The input to the transformation functions is a String, and the output should be the transformed String.
preprocess is sent the raw input that is read from the source file (modulo the default line ending transformation). postprocess is given different things depending on the output: For markdown and script output postprocess is given the content String just before writing it to the output file, but for notebook output postprocess is given the dictionary representing the notebook, since, in general, this is more useful.
As an example, lets say we want to splice the date of generation into the output. We could of course update our source file before generating the docs, but we could instead use a preprocess function that splices the date into the source for us. Consider the following source file:
# # Example
+5. Custom pre- and post-processing · Literate.jl
5. Custom pre- and post-processing
Since all packages are different, and may have different demands on how to create a nice example for the documentation it is important that the package maintainer does not feel limited by the by default provided syntax that this package offers. While you can generally come a long way by utilizing line filtering there might be situations where you need to manually hook into the generation and change things. In Literate this is done by letting the user supply custom pre- and post-processing functions that may do transformation of the content.
All of the generators (Literate.markdown, Literate.notebook and Literate.script) accepts preprocess and postprocess keyword arguments. The default "transformation" is the identity function. The input to the transformation functions is a String, and the output should be the transformed String.
preprocess is sent the raw input that is read from the source file (modulo the default line ending transformation). postprocess is given different things depending on the output: For markdown and script output postprocess is given the content String just before writing it to the output file, but for notebook output postprocess is given the dictionary representing the notebook, since, in general, this is more useful.
As an example, lets say we want to splice the date of generation into the output. We could of course update our source file before generating the docs, but we could instead use a preprocess function that splices the date into the source for us. Consider the following source file:
# # Example
# This example was generated DATEOFTODAY
x = 1 // 3
where DATEOFTODAY is a placeholder, to make it easier for our preprocess function to find the location. Now, lets define the preprocess function, for example
function update_date(content)
diff --git a/latest/documenter.html b/latest/documenter.html
index 56f688d..7a431f3 100644
--- a/latest/documenter.html
+++ b/latest/documenter.html
@@ -1,5 +1,5 @@
-6. Interaction with Documenter.jl · Literate.jl
6. Interaction with Documenter.jl
Literate can be used for any purpose, it spits out regular markdown files, and notebooks. Typically, though, these files will be used to render documentation for your package. The generators (Literate.markdown, Literate.notebook and Literate.script) supports a keyword argument documenter that lets the generator perform some extra things, keeping in mind that the source code have been written with Documenter.jl in mind. So lets take a look at what will happen if we set documenter = true:
Literate.markdown:
- The default code fence will change from
```julia
+6. Interaction with Documenter.jl · Literate.jl
6. Interaction with Documenter.jl
Literate can be used for any purpose, it spits out regular markdown files, and notebooks. Typically, though, these files will be used to render documentation for your package. The generators (Literate.markdown, Literate.notebook and Literate.script) supports a keyword argument documenter that lets the generator perform some extra things, keeping in mind that the source code have been written with Documenter.jl in mind. So lets take a look at what will happen if we set documenter = true:
Literate.markdown:
- The default code fence will change from
```julia
# code
```
to Documenters @example blocks:```@examples $(name)
# code
diff --git a/latest/fileformat.html b/latest/fileformat.html
index a1d454a..8a24e34 100644
--- a/latest/fileformat.html
+++ b/latest/fileformat.html
@@ -1,5 +1,5 @@
-2. File Format · Literate.jl
2. File Format
The source file format for Literate is a regular, commented, julia (.jl) scripts. The idea is that the scripts also serve as documentation on their own and it is also simple to include them in the test-suite, with e.g. include, to make sure the examples stay up do date with other changes in your package.
2.1. Syntax
The basic syntax is simple:
- lines starting with
# are treated as markdown, - all other lines are treated as julia code.
Leading whitespace is allowed before #, but it will be removed when generating the output. Since #-lines is treated as markdown we can not use that for regular julia comments, for this you can instead use ##, which will render as # in the output.
Lets look at a simple example:
# # Rational numbers
+2. File Format · Literate.jl
2. File Format
The source file format for Literate is a regular, commented, julia (.jl) scripts. The idea is that the scripts also serve as documentation on their own and it is also simple to include them in the test-suite, with e.g. include, to make sure the examples stay up do date with other changes in your package.
2.1. Syntax
The basic syntax is simple:
- lines starting with
# are treated as markdown, - all other lines are treated as julia code.
Leading whitespace is allowed before #, but it will be removed when generating the output. Since #-lines is treated as markdown we can not use that for regular julia comments, for this you can instead use ##, which will render as # in the output.
Lets look at a simple example:
# # Rational numbers
#
# In julia rational numbers can be constructed with the `//` operator.
# Lets define two rational numbers, `x` and `y`:
diff --git a/latest/generated/example.html b/latest/generated/example.html
index 35104b6..ac7a575 100644
--- a/latest/generated/example.html
+++ b/latest/generated/example.html
@@ -1,5 +1,5 @@
-7. Example · Literate.jl
7. Example
This is an example generated with Literate based on this source file: example.jl. You are seeing the html-output which Documenter have generated based on a markdown file generated with Literate. The corresponding notebook can be found here: example.ipynb, and the plain script output can be found here: example.jl.
It is recommended to have the source file available when reading this, to better understand how the syntax in the source file corresponds to the output you are seeing.
Basic syntax
The basic syntax for Literate is simple, lines starting with # is interpreted as markdown, and all the other lines are interpreted as code. Here is some code:
x = 1//3
+7. Example · Literate.jl
7. Example
This is an example generated with Literate based on this source file: example.jl. You are seeing the html-output which Documenter have generated based on a markdown file generated with Literate. The corresponding notebook can be found here: example.ipynb, and the plain script output can be found here: example.jl.
It is recommended to have the source file available when reading this, to better understand how the syntax in the source file corresponds to the output you are seeing.
Basic syntax
The basic syntax for Literate is simple, lines starting with # is interpreted as markdown, and all the other lines are interpreted as code. Here is some code:
x = 1//3
y = 2//5
2//5
In markdown sections we can use markdown syntax. For example, we can write text in italic font, text in bold font and use links.
It is possible to filter out lines depending on the output using the #md, #nb, #jl and #src tags (see Filtering Lines):
- This line starts with
#md and is thus only visible in the markdown output.
The source file is parsed in chunks of markdown and code. Starting a line with #- manually inserts a chunk break. For example, if we want to display the output of the following operations we may insert #- in between. These two code blocks will now end up in different @example-blocks in the markdown output, and two different notebook cells in the notebook output.
x + y
11//15
x * y
2//15
Output Capturing
Code chunks are by default placed in Documenter @example blocks in the generated markdown. This means that the output will be captured in a block when Documenter is building the docs. In notebooks the output is captured in output cells, if the execute keyword argument is set to true. Output to stdout/stderr is also captured.
function foo()
println("This string is printed to stdout.")
return [1, 2, 3, 4]
@@ -17,114 +17,114 @@ y2 = cos.(x)
plot(x, [y1, y2])
diff --git a/latest/generated/example.ipynb b/latest/generated/example.ipynb
index 97d9bc9..282cb69 100644
--- a/latest/generated/example.ipynb
+++ b/latest/generated/example.ipynb
@@ -178,18 +178,6 @@
},
{
"outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "┌ Warning: `Array{T}(d::NTuple{N, Int}) where {T, N}` is deprecated, use `Array{T}(undef, d)` instead.\n",
- "│ caller = plot_color(::Array{ColorTypes.RGB{FixedPointNumbers.Normed{UInt8,8}},1}) at colors.jl:24\n",
- "└ @ PlotUtils /home/travis/.julia/packages/PlotUtils/xQ9vp/src/colors.jl:24\n",
- "┌ Warning: `a::Number + b::AbstractArray` is deprecated, use `a .+ b` instead.\n",
- "│ caller = get_zvalues(::Int64) at color_utils.jl:104\n",
- "└ @ PlotUtils /home/travis/.julia/packages/PlotUtils/xQ9vp/src/color_utils.jl:104\n"
- ]
- },
{
"output_type": "execute_result",
"data": {
@@ -199,114 +187,114 @@
"\n",
"\n"
@@ -536,114 +524,114 @@
"\n",
"\n"
@@ -952,11 +940,11 @@
"file_extension": ".jl",
"mimetype": "application/julia",
"name": "julia",
- "version": "0.7.0-beta2.204"
+ "version": "1.0.0"
},
"kernelspec": {
- "name": "julia-0.7",
- "display_name": "Julia 0.7.0-beta2.204",
+ "name": "julia-1.0",
+ "display_name": "Julia 1.0.0",
"language": "julia"
}
},
diff --git a/latest/generated/notebook.ipynb b/latest/generated/notebook.ipynb
index 7f4ba90..44380f6 100644
--- a/latest/generated/notebook.ipynb
+++ b/latest/generated/notebook.ipynb
@@ -88,11 +88,11 @@
"file_extension": ".jl",
"mimetype": "application/julia",
"name": "julia",
- "version": "0.7.0-beta2.204"
+ "version": "1.0.0"
},
"kernelspec": {
- "name": "julia-0.7",
- "display_name": "Julia 0.7.0-beta2.204",
+ "name": "julia-1.0",
+ "display_name": "Julia 1.0.0",
"language": "julia"
}
},
diff --git a/latest/index.html b/latest/index.html
index 19e53bb..89d2480 100644
--- a/latest/index.html
+++ b/latest/index.html
@@ -1,2 +1,2 @@
-1. Introduction · Literate.jl
1. Introduction
Welcome to the documentation for Literate – a simplistic package for Literate Programming.
What?
Literate is a package that generates markdown pages (for e.g. Documenter.jl), and Jupyter notebooks, from the same source file. There is also an option to "clean" the source from all metadata, and produce a pure Julia script.
The main design goal is simplicity. It should be simple to use, and the syntax should be simple. In short, all you have to do is to write a commented julia script!
The public interface consists mainly of three functions, all of which take the same script file as input, but generate different output:
Literate.markdown: generates a markdown fileLiterate.notebook: generates an (optionally executed) notebookLiterate.script: generates a plain script file, removing all metadata and special syntax.
Why?
Examples are (probably) the best way to showcase your awesome package, and examples are often the best way for a new user to learn how to use it. It is therefore important that the documentation of your package contains examples for users to read and study. However, people are different, and we all prefer different ways of trying out a new package. Some people wants to RTFM, others want to explore the package interactively in, for example, a notebook, and some people wants to study the source code. The aim of Literate is to make it easy to give the user all of these options, while still keeping maintenance to a minimum.
It is quite common that packages have "example notebooks" to showcase the package. Notebooks are great for showcasing a package, but they are not so great with version control, like git. The reason being that a notebook is a very "rich" format since it contains output and other metadata. Changes to the notebook thus result in large diffs, which makes it harder to review the actual changes.
It is also common that packages include examples in the documentation, for example by using Documenter.jl @example-blocks. This is also great, but it is not quite as interactive as a notebook, for the users who prefer that.
Literate tries to solve the problems above by creating the output as a part of the doc build. Literate generates the output based on a single source file which makes it easier to maintain, test, and keep the manual and your example notebooks in sync.
1. Introduction · Literate.jl
1. Introduction
Welcome to the documentation for Literate – a simplistic package for Literate Programming.
What?
Literate is a package that generates markdown pages (for e.g. Documenter.jl), and Jupyter notebooks, from the same source file. There is also an option to "clean" the source from all metadata, and produce a pure Julia script.
The main design goal is simplicity. It should be simple to use, and the syntax should be simple. In short, all you have to do is to write a commented julia script!
The public interface consists mainly of three functions, all of which take the same script file as input, but generate different output:
Literate.markdown: generates a markdown fileLiterate.notebook: generates an (optionally executed) notebookLiterate.script: generates a plain script file, removing all metadata and special syntax.
Why?
Examples are (probably) the best way to showcase your awesome package, and examples are often the best way for a new user to learn how to use it. It is therefore important that the documentation of your package contains examples for users to read and study. However, people are different, and we all prefer different ways of trying out a new package. Some people wants to RTFM, others want to explore the package interactively in, for example, a notebook, and some people wants to study the source code. The aim of Literate is to make it easy to give the user all of these options, while still keeping maintenance to a minimum.
It is quite common that packages have "example notebooks" to showcase the package. Notebooks are great for showcasing a package, but they are not so great with version control, like git. The reason being that a notebook is a very "rich" format since it contains output and other metadata. Changes to the notebook thus result in large diffs, which makes it harder to review the actual changes.
It is also common that packages include examples in the documentation, for example by using Documenter.jl @example-blocks. This is also great, but it is not quite as interactive as a notebook, for the users who prefer that.
Literate tries to solve the problems above by creating the output as a part of the doc build. Literate generates the output based on a single source file which makes it easier to maintain, test, and keep the manual and your example notebooks in sync.
4. Output Formats · Literate.jl
4. Output Formats
When the source is parsed, and have been processed it is time to render the output. We will consider the following source snippet:
# # Rational numbers
+4. Output Formats · Literate.jl
4. Output Formats
When the source is parsed, and have been processed it is time to render the output. We will consider the following source snippet:
# # Rational numbers
#
# In julia rational numbers can be constructed with the `//` operator.
# Lets define two rational numbers, `x` and `y`:
@@ -31,8 +31,8 @@ When adding `x` and `y` together we obtain a new rational number:
```@example name
z = x + y
-```
We note that lines starting with # are printed as regular markdown, and the code lines have been wrapped in @example blocks. We also note that an @meta block have been added, that sets the EditURL variable. This is used by Documenter to redirect the "Edit on GitHub" link for the page, see Interaction with Documenter.
Some of the output rendering can be controlled with keyword arguments to Literate.markdown:
Literate.markdown — Function.Literate.markdown(inputfile, outputdir; kwargs...)
Generate a markdown file from inputfile and write the result to the directoryoutputdir.
Keyword arguments:
name: name of the output file, excluding .md. name is also used to name all the @example blocks, and to replace @__NAME__. Defaults to the filename of inputfile.preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.documenter: boolean that tells if the output is intended to use with Documenter.jl. Defaults to true. See the the manual section on Interaction with Documenter.codefence: A Pair of opening and closing code fence. Defaults to"```@example $(name)" => "```"
if documenter = true and"```julia" => "```"
if documenter = false.
source4.2. Notebook Output
The (default) notebook output of the source snippet can be seen here: notebook.ipynb.
We note that lines starting with # are placed in markdown cells, and the code lines have been placed in code cells. By default the notebook is also executed and output cells populated. The current working directory is set to the specified output directory the notebook is executed. Some of the output rendering can be controlled with keyword arguments to Literate.notebook:
Literate.notebook — Function.Literate.notebook(inputfile, outputdir; kwargs...)
Generate a notebook from inputfile and write the result to outputdir.
Keyword arguments:
name: name of the output file, excluding .ipynb. name is also used to replace @__NAME__. Defaults to the filename of inputfile.preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.execute: a boolean deciding if the generated notebook should also be executed or not. Defaults to true. The current working directory is set to outputdir when executing the notebook.documenter: boolean that says if the source contains Documenter.jl specific things to filter out during notebook generation. Defaults to true. See the the manual section on Interaction with Documenter.
source4.3. Script Output
The (default) script output of the source snippet above is as follows
x = 1//3
+```
We note that lines starting with # are printed as regular markdown, and the code lines have been wrapped in @example blocks. We also note that an @meta block have been added, that sets the EditURL variable. This is used by Documenter to redirect the "Edit on GitHub" link for the page, see Interaction with Documenter.
Some of the output rendering can be controlled with keyword arguments to Literate.markdown:
Literate.markdown — Function.Literate.markdown(inputfile, outputdir; kwargs...)
Generate a markdown file from inputfile and write the result to the directoryoutputdir.
Keyword arguments:
name: name of the output file, excluding .md. name is also used to name all the @example blocks, and to replace @__NAME__. Defaults to the filename of inputfile.preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.documenter: boolean that tells if the output is intended to use with Documenter.jl. Defaults to true. See the the manual section on Interaction with Documenter.codefence: A Pair of opening and closing code fence. Defaults to"```@example $(name)" => "```"
if documenter = true and"```julia" => "```"
if documenter = false.
source4.2. Notebook Output
The (default) notebook output of the source snippet can be seen here: notebook.ipynb.
We note that lines starting with # are placed in markdown cells, and the code lines have been placed in code cells. By default the notebook is also executed and output cells populated. The current working directory is set to the specified output directory the notebook is executed. Some of the output rendering can be controlled with keyword arguments to Literate.notebook:
Literate.notebook — Function.Literate.notebook(inputfile, outputdir; kwargs...)
Generate a notebook from inputfile and write the result to outputdir.
Keyword arguments:
name: name of the output file, excluding .ipynb. name is also used to replace @__NAME__. Defaults to the filename of inputfile.preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.execute: a boolean deciding if the generated notebook should also be executed or not. Defaults to true. The current working directory is set to outputdir when executing the notebook.documenter: boolean that says if the source contains Documenter.jl specific things to filter out during notebook generation. Defaults to true. See the the manual section on Interaction with Documenter.
source4.3. Script Output
The (default) script output of the source snippet above is as follows
x = 1//3
y = 2//5
-z = x + y
We note that lines starting with # are removed and only the code lines have been kept. Some of the output rendering can be controlled with keyword arguments to Literate.script:
Literate.script — Function.Literate.script(inputfile, outputdir; kwargs...)
Generate a plain script file from inputfile and write the result to outputdir.
Keyword arguments:
name: name of the output file, excluding .jl. name is also used to replace @__NAME__. Defaults to the filename of inputfile.preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.documenter: boolean that says if the source contains Documenter.jl specific things to filter out during script generation. Defaults to true. See the the manual section on Interaction with Documenter.keep_comments: boolean that, if set to true, keeps markdown lines as comments in the output script. Defaults to false.
source
We note that lines starting with # are removed and only the code lines have been kept. Some of the output rendering can be controlled with keyword arguments to Literate.script:
Literate.script — Function.Literate.script(inputfile, outputdir; kwargs...)
Generate a plain script file from inputfile and write the result to outputdir.
Keyword arguments:
name: name of the output file, excluding .jl. name is also used to replace @__NAME__. Defaults to the filename of inputfile.preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.documenter: boolean that says if the source contains Documenter.jl specific things to filter out during script generation. Defaults to true. See the the manual section on Interaction with Documenter.keep_comments: boolean that, if set to true, keeps markdown lines as comments in the output script. Defaults to false.
source3. Processing pipeline · Literate.jl
3. Processing pipeline
The generation of output follows the same pipeline for all output formats:
3.1. Pre-processing
The first step is pre-processing of the input file. The file is read to a String. The first processing step is to apply the user specified pre-processing function, see Custom pre- and post-processing.
The next step is to perform all of the built-in default replacements. CRLF style line endings ("\r\n") are replaced with LF line endings ("\n") to simplify internal processing. Next, line filtering is performed, see Filtering Lines, meaning that lines starting with #md, #nb or #jl are handled (either just the token itself is removed, or the full line, depending on the output target). The last pre-processing step is to expand the convenience "macros" described in Default Replacements is expanded.
3.2. Parsing
After the preprocessing the file is parsed. The first step is to categorize each line and mark them as either markdown or code according to the rules described in the Syntax section. Lets consider the example from the previous section with each line categorized:
# # Rational numbers <- markdown
+3. Processing pipeline · Literate.jl
3. Processing pipeline
The generation of output follows the same pipeline for all output formats:
3.1. Pre-processing
The first step is pre-processing of the input file. The file is read to a String. The first processing step is to apply the user specified pre-processing function, see Custom pre- and post-processing.
The next step is to perform all of the built-in default replacements. CRLF style line endings ("\r\n") are replaced with LF line endings ("\n") to simplify internal processing. Next, line filtering is performed, see Filtering Lines, meaning that lines starting with #md, #nb or #jl are handled (either just the token itself is removed, or the full line, depending on the output target). The last pre-processing step is to expand the convenience "macros" described in Default Replacements is expanded.
3.2. Parsing
After the preprocessing the file is parsed. The first step is to categorize each line and mark them as either markdown or code according to the rules described in the Syntax section. Lets consider the example from the previous section with each line categorized:
# # Rational numbers <- markdown
# <- markdown
# In julia rational numbers can be constructed with the `//` operator. <- markdown
# Lets define two rational numbers, `x` and `y`: <- markdown
diff --git a/latest/search.html b/latest/search.html
index 674f80b..67a457d 100644
--- a/latest/search.html
+++ b/latest/search.html
@@ -1,2 +1,2 @@
-Search · Literate.jl
Search
Number of results: loading...
Search · Literate.jl
Search
Number of results: loading...