diff --git a/dev/assets/documenter.js b/dev/assets/documenter.js index 15dc682..2a1aae1 100644 --- a/dev/assets/documenter.js +++ b/dev/assets/documenter.js @@ -1,15 +1,15 @@ // Generated by Documenter.jl requirejs.config({ paths: { - 'highlight-julia': 'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.15.10/languages/julia.min', + 'highlight-julia': 'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.5.0/languages/julia.min', 'headroom': 'https://cdnjs.cloudflare.com/ajax/libs/headroom/0.10.3/headroom.min', 'jqueryui': 'https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.12.1/jquery-ui.min', 'katex-auto-render': 'https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.11.1/contrib/auto-render.min', 'jquery': 'https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.min', 'headroom-jquery': 'https://cdnjs.cloudflare.com/ajax/libs/headroom/0.10.3/jQuery.headroom.min', 'katex': 'https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.11.1/katex.min', - 'highlight': 'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.15.10/highlight.min', - 'highlight-julia-repl': 'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/9.15.10/languages/julia-repl.min', + 'highlight': 'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.5.0/highlight.min', + 'highlight-julia-repl': 'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.5.0/languages/julia-repl.min', }, shim: { "highlight-julia": { @@ -65,7 +65,7 @@ $(document).ready(function() { }) //////////////////////////////////////////////////////////////////////////////// -require(['jquery', 'highlight', 'highlight-julia', 'highlight-julia-repl'], function($, hljs) { +require(['jquery', 'highlight', 'highlight-julia', 'highlight-julia-repl'], function($) { $(document).ready(function() { hljs.initHighlighting(); }) diff --git a/dev/customprocessing/index.html b/dev/customprocessing/index.html index 95530f3..00f6a04 100644 --- a/dev/customprocessing/index.html +++ b/dev/customprocessing/index.html @@ -26,4 +26,4 @@ include("file2.jl")

Let's say we have saved this fi end return str end

(of course replace included with your respective files)

Finally, you simply pass this function to e.g. Literate.markdown as

Literate.markdown("examples.jl", "path/to/save/markdown";
-                  name = "markdown_file_name", preprocess = replace_includes)

and you will see that in the final output file (here markdown_file_name.md) the include statements are replaced with the actual code to be included!

This approach is used for generating the examples in the documentation of the TimeseriesPrediction.jl package. The example files, included together in the stexamples.jl file, are processed by literate via this make.jl file to generate the markdown and code cells of the documentation.

+ name = "markdown_file_name", preprocess = replace_includes)

and you will see that in the final output file (here markdown_file_name.md) the include statements are replaced with the actual code to be included!

This approach is used for generating the examples in the documentation of the TimeseriesPrediction.jl package. The example files, included together in the stexamples.jl file, are processed by literate via this make.jl file to generate the markdown and code cells of the documentation.

diff --git a/dev/documenter/index.html b/dev/documenter/index.html index cd4370d..8866fc0 100644 --- a/dev/documenter/index.html +++ b/dev/documenter/index.html @@ -1,12 +1,12 @@ -6. Interaction with Documenter.jl · Literate.jl
Edit on GitHub

6. Interaction with Documenter.jl

Literate can be used for any purpose, it spits out regular markdown files, and notebooks. Typically, though, these files will be used to render documentation for your package. The generators (Literate.markdown, Literate.notebook and Literate.script) supports a keyword argument documenter that lets the generator perform some extra things, keeping in mind that the source code have been written with Documenter.jl in mind. So let's take a look at what will happen if we set documenter = true:

Literate.markdown:

Literate.script:

  • Documenter style @refs and @id will be removed. This means that you can use @ref and @id in the source file without them leaking to the script.
diff --git a/dev/fileformat/index.html b/dev/fileformat/index.html index ef64d2f..04fdbfc 100644 --- a/dev/fileformat/index.html +++ b/dev/fileformat/index.html @@ -25,4 +25,4 @@ This is also markdown. #md # Literate.notebook #md # Literate.markdown #md # ```

The lines in the example above would be filtered out in the preprocessing step, unless we are generating a markdown file. When generating a markdown file we would simple remove the leading #md from the lines. Beware that the space after the tag is also removed.

The #src token can also be placed at the end of a line. This is to make it possible to have code lines exclusive to the source code, and not just comment lines. For example, if the source file is included in the test suite we might want to add a @test at the end without this showing up in the outputs:

using Test                      #src
-@test result == expected_result #src

2.3. Default Replacements

The following convenience "macros"/source placeholders are always expanded:

Literate 2.1

GitHub Actions support for the macros above requires at least Literate version 2.1.

Literate 2.2

GitLab CI support for the macros above requires at least Literate version 2.2.

+@test result == expected_result #src

2.3. Default Replacements

The following convenience "macros"/source placeholders are always expanded:

Literate 2.1

GitHub Actions support for the macros above requires at least Literate version 2.1.

Literate 2.2

GitLab CI support for the macros above requires at least Literate version 2.2.

diff --git a/dev/generated/example.ipynb b/dev/generated/example.ipynb index 13cfead..636f274 100644 --- a/dev/generated/example.ipynb +++ b/dev/generated/example.ipynb @@ -278,87 +278,87 @@ "\n", "\n", "\n", - " \n", + " \n", " \n", " \n", "\n", - "\n", "\n", - " \n", + " \n", " \n", " \n", "\n", - "\n", "\n", - " \n", + " \n", " \n", " \n", "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n" + "\n" ], "image/svg+xml": [ "\n", "\n", "\n", - " \n", + " \n", " \n", " \n", "\n", - "\n", "\n", - " \n", + " \n", " \n", " \n", "\n", - "\n", "\n", - " \n", + " \n", " \n", " \n", "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n" + "\n" ] }, "metadata": {}, @@ -1041,11 +1041,11 @@ "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", - "version": "1.5.3" + "version": "1.5.4" }, "kernelspec": { "name": "julia-1.5", - "display_name": "Julia 1.5.3", + "display_name": "Julia 1.5.4", "language": "julia" } }, diff --git a/dev/generated/example/index.html b/dev/generated/example/index.html index ffe219a..c2aef50 100644 --- a/dev/generated/example/index.html +++ b/dev/generated/example/index.html @@ -16,87 +16,87 @@ y2 = cos.(x) plot(x, [y1, y2]) - + - - + - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - +

Custom processing

It is possible to give Literate custom pre- and post-processing functions. For example, here we insert a placeholder value y = 321 in the source, and use a preprocessing function that replaces it with y = 321 in the rendered output.

x = 123
123

In this case the preprocessing function is defined by

function pre(s::String)
     s = replace(s, "x = 123" => "y = 321")
     return s
-end
pre (generic function with 1 method)

Documenter.jl interaction

In the source file it is possible to use Documenter.jl style references, such as @ref and @id. These will be filtered out in the notebook output. For example, here is a link, but it is only visible as a link if you are reading the markdown output. We can also use equations:

\[\int_\Omega \nabla v \cdot \nabla u\ \mathrm{d}\Omega = \int_\Omega v f\ \mathrm{d}\Omega\]

using Documenters math syntax. Documenters syntax is automatically changed to \begin{equation} ... \end{equation} in the notebook output to display correctly.

This page was generated using Literate.jl.

+end
pre (generic function with 1 method)

Documenter.jl interaction

In the source file it is possible to use Documenter.jl style references, such as @ref and @id. These will be filtered out in the notebook output. For example, here is a link, but it is only visible as a link if you are reading the markdown output. We can also use equations:

\[\int_\Omega \nabla v \cdot \nabla u\ \mathrm{d}\Omega = \int_\Omega v f\ \mathrm{d}\Omega\]

using Documenters math syntax. Documenters syntax is automatically changed to \begin{equation} ... \end{equation} in the notebook output to display correctly.

This page was generated using Literate.jl.

diff --git a/dev/generated/name/index.html b/dev/generated/name/index.html index e7e2c53..cc5a439 100644 --- a/dev/generated/name/index.html +++ b/dev/generated/name/index.html @@ -1,2 +1,2 @@ -Rational numbers · Literate.jl
Edit on GitHub

Rational numbers

In julia rational numbers can be constructed with the // operator. Lets define two rational numbers, x and y:

x = 1//3
1//3
y = 2//5
2//5

When adding x and y together we obtain a new rational number:

z = x + y
11//15
+Rational numbers · Literate.jl
Edit on GitHub

Rational numbers

In julia rational numbers can be constructed with the // operator. Lets define two rational numbers, x and y:

x = 1//3
1//3
y = 2//5
2//5

When adding x and y together we obtain a new rational number:

z = x + y
11//15
diff --git a/dev/generated/notebook.ipynb b/dev/generated/notebook.ipynb index 9a84d78..6cb1df0 100644 --- a/dev/generated/notebook.ipynb +++ b/dev/generated/notebook.ipynb @@ -87,11 +87,11 @@ "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", - "version": "1.5.3" + "version": "1.5.4" }, "kernelspec": { "name": "julia-1.5", - "display_name": "Julia 1.5.3", + "display_name": "Julia 1.5.4", "language": "julia" } }, diff --git a/dev/index.html b/dev/index.html index d3ad604..19a7c49 100644 --- a/dev/index.html +++ b/dev/index.html @@ -1,2 +1,2 @@ -1. Introduction · Literate.jl
Edit on GitHub

1. Introduction

Welcome to the documentation for Literate – a simplistic package for Literate Programming.

What?

Literate is a package that generates markdown pages (for e.g. Documenter.jl), and Jupyter notebooks, from the same source file. There is also an option to "clean" the source from all metadata, and produce a pure Julia script.

The main design goal is simplicity. It should be simple to use, and the syntax should be simple. In short, all you have to do is to write a commented julia script!

The public interface consists of three functions, all of which take the same script file as input, but generate different output:

  • Literate.markdown generates a markdown file. Code snippets can be executed and the results included in the output.
  • Literate.notebook generates a notebook. Code snippets can be executed and the results included in the output.
  • Literate.script generates a plain script file scrubbed from all metadata and special syntax.

Why?

Examples are (probably) the best way to showcase your awesome package, and examples are often the best way for a new user to learn how to use it. It is therefore important that the documentation of your package contains examples for users to read and study. However, people are different, and we all prefer different ways of trying out a new package. Some people wants to RTFM, others want to explore the package interactively in, for example, a notebook, and some people wants to study the source code. The aim of Literate is to make it easy to give the user all of these options, while still keeping maintenance to a minimum.

It is quite common that packages have "example notebooks" to showcase the package. Notebooks are great for showcasing a package, but they are not so great with version control, like git. The reason being that a notebook is a very "rich" format since it contains output and other metadata. Changes to the notebook thus result in large diffs, which makes it harder to review the actual changes.

It is also common that packages include examples in the documentation, for example by using Documenter.jl @example-blocks. This is also great, but it is not quite as interactive as a notebook, for the users who prefer that.

Literate tries to solve the problems above by creating the output as a part of the doc build. Literate generates the output based on a single source file which makes it easier to maintain, test, and keep the manual and your example notebooks in sync.

+1. Introduction · Literate.jl
Edit on GitHub

1. Introduction

Welcome to the documentation for Literate – a simplistic package for Literate Programming.

What?

Literate is a package that generates markdown pages (for e.g. Documenter.jl), and Jupyter notebooks, from the same source file. There is also an option to "clean" the source from all metadata, and produce a pure Julia script.

The main design goal is simplicity. It should be simple to use, and the syntax should be simple. In short, all you have to do is to write a commented julia script!

The public interface consists of three functions, all of which take the same script file as input, but generate different output:

  • Literate.markdown generates a markdown file. Code snippets can be executed and the results included in the output.
  • Literate.notebook generates a notebook. Code snippets can be executed and the results included in the output.
  • Literate.script generates a plain script file scrubbed from all metadata and special syntax.

Why?

Examples are (probably) the best way to showcase your awesome package, and examples are often the best way for a new user to learn how to use it. It is therefore important that the documentation of your package contains examples for users to read and study. However, people are different, and we all prefer different ways of trying out a new package. Some people wants to RTFM, others want to explore the package interactively in, for example, a notebook, and some people wants to study the source code. The aim of Literate is to make it easy to give the user all of these options, while still keeping maintenance to a minimum.

It is quite common that packages have "example notebooks" to showcase the package. Notebooks are great for showcasing a package, but they are not so great with version control, like git. The reason being that a notebook is a very "rich" format since it contains output and other metadata. Changes to the notebook thus result in large diffs, which makes it harder to review the actual changes.

It is also common that packages include examples in the documentation, for example by using Documenter.jl @example-blocks. This is also great, but it is not quite as interactive as a notebook, for the users who prefer that.

Literate tries to solve the problems above by creating the output as a part of the doc build. Literate generates the output based on a single source file which makes it easier to maintain, test, and keep the manual and your example notebooks in sync.

diff --git a/dev/outputformats/index.html b/dev/outputformats/index.html index 86d3bb0..c8ef08d 100644 --- a/dev/outputformats/index.html +++ b/dev/outputformats/index.html @@ -36,7 +36,7 @@ x = 1//3 ``` ``` 1//3 -```

In this example the output is just plain text. However, if the resulting value of the code block can be displayed as an image (png or jpeg) Literate will include the image representation of the output.

Note

Since Documenter executes and captures results of @example block it is not necessary to use execute=true for markdown output that is meant to be used as input to Documenter.

Literate 2.4

Code execution of markdown output requires at least Literate version 2.4.

See the section about Configuration for more information about how to configure the behavior and resulting output of Literate.markdown.

Literate.markdown — Function
Literate.markdown(inputfile, outputdir=pwd(); config::Dict=Dict(), kwargs...)

Generate a markdown file from inputfile and write the result to the directory outputdir.

Literate 2.5

Default output directory pwd requires at least Literate version 2.5.

See the manual section on Configuration for documentation of possible configuration with config and other keyword arguments.

source

4.2. Notebook Output

Notebook output is generated by Literate.notebook. The (default) notebook output of the source snippet can be seen here: notebook.ipynb.

We note that lines starting with # are placed in markdown cells, and the code lines have been placed in code cells. By default the notebook is also executed and output cells populated. The current working directory is set to the specified output directory the notebook is executed.

See the section about Configuration for how to configure the behavior and resulting output of Literate.notebook.

Literate.notebook — Function
Literate.notebook(inputfile, outputdir=pwd(); config::Dict=Dict(), kwargs...)

Generate a notebook from inputfile and write the result to outputdir.

Literate 2.5

Default output directory pwd requires at least Literate version 2.5.

See the manual section on Configuration for documentation of possible configuration with config and other keyword arguments.

source

Notebook metadata

Jupyter notebook cells (both code cells and markdown cells) can contain metadata. This is enabled in Literate by the %% token, similar to Jupytext. The format is as follows

%% optional ignored text [type] {optional metadata JSON}

Cell metadata can, for example, be used for nbgrader and the reveal.js notebook extension RISE.

The following would create a 3 slide deck with RISE:

#nb # %% A slide [markdown] {"slideshow": {"slide_type": "slide"}}
+```

In this example the output is just plain text. However, if the resulting value of the code block can be displayed as an image (png or jpeg) Literate will include the image representation of the output.

Note

Since Documenter executes and captures results of @example block it is not necessary to use execute=true for markdown output that is meant to be used as input to Documenter.

Literate 2.4

Code execution of markdown output requires at least Literate version 2.4.

See the section about Configuration for more information about how to configure the behavior and resulting output of Literate.markdown.

Literate.markdown — Function
Literate.markdown(inputfile, outputdir=pwd(); config::Dict=Dict(), kwargs...)

Generate a markdown file from inputfile and write the result to the directory outputdir.

Literate 2.5

Default output directory pwd requires at least Literate version 2.5.

See the manual section on Configuration for documentation of possible configuration with config and other keyword arguments.

source

4.2. Notebook Output

Notebook output is generated by Literate.notebook. The (default) notebook output of the source snippet can be seen here: notebook.ipynb.

We note that lines starting with # are placed in markdown cells, and the code lines have been placed in code cells. By default the notebook is also executed and output cells populated. The current working directory is set to the specified output directory the notebook is executed.

See the section about Configuration for how to configure the behavior and resulting output of Literate.notebook.

Literate.notebook — Function
Literate.notebook(inputfile, outputdir=pwd(); config::Dict=Dict(), kwargs...)

Generate a notebook from inputfile and write the result to outputdir.

Literate 2.5

Default output directory pwd requires at least Literate version 2.5.

See the manual section on Configuration for documentation of possible configuration with config and other keyword arguments.

source

Notebook metadata

Jupyter notebook cells (both code cells and markdown cells) can contain metadata. This is enabled in Literate by the %% token, similar to Jupytext. The format is as follows

%% optional ignored text [type] {optional metadata JSON}

Cell metadata can, for example, be used for nbgrader and the reveal.js notebook extension RISE.

The following would create a 3 slide deck with RISE:

#nb # %% A slide [markdown] {"slideshow": {"slide_type": "slide"}}
 # # Some title
 #
 # We're using `#nb` so the metadata is only included in notebook output
@@ -50,4 +50,4 @@ y = 2//5
 
 y = 2//5
 
-z = x + y

We note that lines starting with # are removed and only the code lines have been kept.

See the section about Configuration for how to configure the behavior and resulting output of Literate.script.

Literate.script — Function
Literate.script(inputfile, outputdir=pwd(); config::Dict=Dict(), kwargs...)

Generate a plain script file from inputfile and write the result to outputdir.

Literate 2.5

Default output directory pwd requires at least Literate version 2.5.

See the manual section on Configuration for documentation of possible configuration with config and other keyword arguments.

source

4.4. Configuration

The behavior of Literate.markdown, Literate.notebook and Literate.script can be configured by keyword arguments. There are two ways to do this; pass config::Dict as a keyword argument, or pass individual keyword arguments.

Literate 2.2

Passing configuration as a dictionary requires at least Literate version 2.2.

Configuration precedence

Individual keyword arguments takes precedence over the config dictionary, so for e.g. Literate.markdown(...; config = Dict("name" => "hello"), name = "world") the resulting configuration for name will be "world". Both individual keyword arguments and the config dictionary takes precedence over the default.

Available configurations with description and default values are given in the reference for Literate.DEFAULT_CONFIGURATION just below.

Literate.DEFAULT_CONFIGURATION — Constant
DEFAULT_CONFIGURATION

Default configuration for Literate.markdown, Literate.notebook and Literate.script which is used for everything not specified by the user. See the manual section about Configuration for more information.

Configuration keyDescriptionDefault valueComment
nameName of the output file (excluding file extension).filename(inputfile)
preprocessCustom preprocessing function mapping String to String.identitySee Custom pre- and post-processing.
postprocessCustom preprocessing function mapping String to String.identitySee Custom pre- and post-processing.
documenterBoolean signaling that the source contains Documenter.jl elements.trueSee Interaction with Documenter.
creditBoolean for controlling the addition of This file was generated with Literate.jl ... to the bottom of the page. If you find Literate.jl useful then feel free to keep this.true
keep_commentsWhen true, keeps markdown lines as comments in the output script.falseOnly applicable for Literate.script.
executeWhether to execute and capture the output.true (notebook), false (markdown)Only applicable for Literate.notebook and Literate.markdown. For markdown this requires at least Literate 2.4.
codefencePair containing opening and closing fence for wrapping code blocks."```julia" => "```"If documenter is true the default is "```@example"=>"```".
devurlURL for "in-development" docs."dev"See Documenter docs. Unused if repo_root_url/nbviewer_root_url/binder_root_url are set.
repo_root_urlURL to the root of the repository.-Determined automatically on Travis CI, GitHub Actions and GitLab CI. Used for @__REPO_ROOT_URL__.
nbviewer_root_urlURL to the root of the repository as seen on nbviewer.-Determined automatically on Travis CI, GitHub Actions and GitLab CI. Used for @__NBVIEWER_ROOT_URL__.
binder_root_urlURL to the root of the repository as seen on mybinder.-Determined automatically on Travis CI, GitHub Actions and GitLab CI. Used for @__BINDER_ROOT_URL__.
repo_root_pathFilepath to the root of the repository.-Determined automatically on Travis CI, GitHub Actions and GitLab CI. Used for computing Documenters EditURL.
source
+z = x + y

We note that lines starting with # are removed and only the code lines have been kept.

See the section about Configuration for how to configure the behavior and resulting output of Literate.script.

Literate.script — Function
Literate.script(inputfile, outputdir=pwd(); config::Dict=Dict(), kwargs...)

Generate a plain script file from inputfile and write the result to outputdir.

Literate 2.5

Default output directory pwd requires at least Literate version 2.5.

See the manual section on Configuration for documentation of possible configuration with config and other keyword arguments.

source

4.4. Configuration

The behavior of Literate.markdown, Literate.notebook and Literate.script can be configured by keyword arguments. There are two ways to do this; pass config::Dict as a keyword argument, or pass individual keyword arguments.

Literate 2.2

Passing configuration as a dictionary requires at least Literate version 2.2.

Configuration precedence

Individual keyword arguments takes precedence over the config dictionary, so for e.g. Literate.markdown(...; config = Dict("name" => "hello"), name = "world") the resulting configuration for name will be "world". Both individual keyword arguments and the config dictionary takes precedence over the default.

Available configurations with description and default values are given in the reference for Literate.DEFAULT_CONFIGURATION just below.

Literate.DEFAULT_CONFIGURATION — Constant
DEFAULT_CONFIGURATION

Default configuration for Literate.markdown, Literate.notebook and Literate.script which is used for everything not specified by the user. See the manual section about Configuration for more information.

Configuration keyDescriptionDefault valueComment
nameName of the output file (excluding file extension).filename(inputfile)
preprocessCustom preprocessing function mapping String to String.identitySee Custom pre- and post-processing.
postprocessCustom preprocessing function mapping String to String.identitySee Custom pre- and post-processing.
documenterBoolean signaling that the source contains Documenter.jl elements.trueSee Interaction with Documenter.
creditBoolean for controlling the addition of This file was generated with Literate.jl ... to the bottom of the page. If you find Literate.jl useful then feel free to keep this.true
keep_commentsWhen true, keeps markdown lines as comments in the output script.falseOnly applicable for Literate.script.
executeWhether to execute and capture the output.true (notebook), false (markdown)Only applicable for Literate.notebook and Literate.markdown. For markdown this requires at least Literate 2.4.
codefencePair containing opening and closing fence for wrapping code blocks."```julia" => "```"If documenter is true the default is "```@example"=>"```".
devurlURL for "in-development" docs."dev"See Documenter docs. Unused if repo_root_url/nbviewer_root_url/binder_root_url are set.
repo_root_urlURL to the root of the repository.-Determined automatically on Travis CI, GitHub Actions and GitLab CI. Used for @__REPO_ROOT_URL__.
nbviewer_root_urlURL to the root of the repository as seen on nbviewer.-Determined automatically on Travis CI, GitHub Actions and GitLab CI. Used for @__NBVIEWER_ROOT_URL__.
binder_root_urlURL to the root of the repository as seen on mybinder.-Determined automatically on Travis CI, GitHub Actions and GitLab CI. Used for @__BINDER_ROOT_URL__.
repo_root_pathFilepath to the root of the repository.-Determined automatically on Travis CI, GitHub Actions and GitLab CI. Used for computing Documenters EditURL.
source
diff --git a/dev/pipeline/index.html b/dev/pipeline/index.html index 303cc3c..6c9bb63 100644 --- a/dev/pipeline/index.html +++ b/dev/pipeline/index.html @@ -1,5 +1,5 @@ -3. Processing pipeline · Literate.jl
Edit on GitHub

3. Processing pipeline

The generation of output follows the same pipeline for all output formats:

  1. Pre-processing
  2. Parsing
  3. Document generation
  4. Post-processing
  5. Writing to file

3.1. Pre-processing

The first step is pre-processing of the input file. The file is read to a String. The first processing step is to apply the user specified pre-processing function, see Custom pre- and post-processing.

The next step is to perform all of the built-in default replacements. CRLF style line endings ("\r\n") are replaced with LF line endings ("\n") to simplify internal processing. Next, line filtering is performed, see Filtering Lines, meaning that lines starting with #md, #nb or #jl are handled (either just the token itself is removed, or the full line, depending on the output target). The last pre-processing step is to expand the convenience "macros" described in Default Replacements is expanded.

3.2. Parsing

After the preprocessing the file is parsed. The first step is to categorize each line and mark them as either markdown or code according to the rules described in the Syntax section. Lets consider the example from the previous section with each line categorized:

# # Rational numbers                                                     <- markdown
+3. Processing pipeline · Literate.jl
Edit on GitHub
3. Processing pipeline
The generation of output follows the same pipeline for all output formats:
  1. Pre-processing
  2. Parsing
  3. Document generation
  4. Post-processing
  5. Writing to file
3.1. Pre-processing
The first step is pre-processing of the input file. The file is read to a String. The first processing step is to apply the user specified pre-processing function, see Custom pre- and post-processing.
The next step is to perform all of the built-in default replacements. CRLF style line endings ("\r\n") are replaced with LF line endings ("\n") to simplify internal processing. Next, line filtering is performed, see Filtering Lines, meaning that lines starting with #md, #nb or #jl are handled (either just the token itself is removed, or the full line, depending on the output target). The last pre-processing step is to expand the convenience "macros" described in Default Replacements is expanded.
3.2. Parsing
After the preprocessing the file is parsed. The first step is to categorize each line and mark them as either markdown or code according to the rules described in the Syntax section. Lets consider the example from the previous section with each line categorized:
# # Rational numbers                                                     <- markdown
 #                                                                        <- markdown
 # In julia rational numbers can be constructed with the `//` operator.   <- markdown
 # Lets define two rational numbers, `x` and `y`:                         <- markdown
@@ -10,7 +10,7 @@ y = 2 // 5                                                               <- c
                                                                          <- code
 # When adding `x` and `y` together we obtain a new rational number:      <- markdown
                                                                          <- code
-z = x + y                                                                <- code
In the next step the lines are grouped into "chunks" of markdown and code. This is done by simply collecting adjacent lines of the same "type" into chunks:
# # Rational numbers                                                     ┐
+z = x + y                                                                <- code
In the next step the lines are grouped into "chunks" of markdown and code. This is done by simply collecting adjacent lines of the same "type" into chunks:
# # Rational numbers                                                     ┐
 #                                                                        │
 # In julia rational numbers can be constructed with the `//` operator.   │ markdown
 # Lets define two rational numbers, `x` and `y`:                         ┘
@@ -29,4 +29,4 @@ x = 1 // 3
 y = 2 // 5
Chunk #3:
When adding `x` and `y` together we obtain a new rational number:
Chunk #4:
z = x + y
It is then up to the Document generation step to decide how these chunks should be treated.
Custom control over chunk splits
Sometimes it is convenient to be able to manually control how the chunks are split. For example, if you want to split a block of code into two, such that they end up in two different @example blocks or notebook cells. The #- token can be used for this purpose. All lines starting with #- are used as "chunk-splitters":
x = 1 // 3
 y = 2 // 5
 #-
-z = x + y
The example above would result in two consecutive code-chunks.
Tip
The rest of the line, after #-, is discarded, so it is possible to use e.g. #------------- as a chunk splitter, which may make the source code more readable.
It is also possible to use #+ as a chunk splitter. The difference between #+ and #- is that #+ enables Documenter's "continued"-blocks, see the Documenter manual.
3.3. Document generation
After the parsing it is time to generate the output. What is done in this step is very different depending on the output target, and it is described in more detail in the Output format sections: Markdown Output, Notebook Output and Script Output. Using the default settings, the following is happening:
  • Markdown output: markdown chunks are printed as-is, code chunks are put inside a code fence (defaults to @example-blocks),
  • Notebook output: markdown chunks are printed in markdown cells, code chunks are put in code cells,
  • Script output: markdown chunks are discarded, code chunks are printed as-is.
3.4. Post-processing
When the document is generated the user, again, has the option to hook-into the generation with a custom post-processing function. The reason is that one might want to change things that are only visible in the rendered document. See Custom pre- and post-processing.
3.5. Writing to file
The last step of the generation is writing to file. The result is written to $(outputdir)/$(name)(.md|.ipynb|.jl) where outputdir is the output directory supplied by the user (for example docs/generated), and name is a user supplied filename. It is recommended to add the output directory to .gitignore since the idea is that the generated documents will be generated as part of the build process rather than beeing files in the repo.

+z = x + y

The example above would result in two consecutive code-chunks.

Tip

The rest of the line, after #-, is discarded, so it is possible to use e.g. #------------- as a chunk splitter, which may make the source code more readable.

It is also possible to use #+ as a chunk splitter. The difference between #+ and #- is that #+ enables Documenter's "continued"-blocks, see the Documenter manual.

3.3. Document generation

After the parsing it is time to generate the output. What is done in this step is very different depending on the output target, and it is described in more detail in the Output format sections: Markdown Output, Notebook Output and Script Output. Using the default settings, the following is happening:

  • Markdown output: markdown chunks are printed as-is, code chunks are put inside a code fence (defaults to @example-blocks),
  • Notebook output: markdown chunks are printed in markdown cells, code chunks are put in code cells,
  • Script output: markdown chunks are discarded, code chunks are printed as-is.

3.4. Post-processing

When the document is generated the user, again, has the option to hook-into the generation with a custom post-processing function. The reason is that one might want to change things that are only visible in the rendered document. See Custom pre- and post-processing.

3.5. Writing to file

The last step of the generation is writing to file. The result is written to $(outputdir)/$(name)(.md|.ipynb|.jl) where outputdir is the output directory supplied by the user (for example docs/generated), and name is a user supplied filename. It is recommended to add the output directory to .gitignore since the idea is that the generated documents will be generated as part of the build process rather than beeing files in the repo.

diff --git a/dev/search/index.html b/dev/search/index.html index 6ab7b59..9ff98e0 100644 --- a/dev/search/index.html +++ b/dev/search/index.html @@ -1,2 +1,2 @@ -Search · Literate.jl

Loading search...

    +Search · Literate.jl

    Loading search...

      diff --git a/dev/tips/index.html b/dev/tips/index.html index 2830cf8..82ea341 100644 --- a/dev/tips/index.html +++ b/dev/tips/index.html @@ -2,4 +2,4 @@ 7. Tips and Tricks · Literate.jl
      Edit on GitHub

      7. Tips and Tricks

      This section lists some tips and tricks that might be useful for using Literate.

      Filesize of generated notebooks

      When Literate executes a notebook the return value, i.e. the result of the last Julia expression in each cell is captured. By default Literate generates multiple renderings of the result in different output formats or MIMEs, just like IJulia.jl does. All of these renderings are embedded in the notebook and it is up to the notebook frontend viewer to select the most appropriate format to show to the user.

      A common example is images, which can often be displayed in multiple formats, e.g. PNG (image/png), SVG (image/svg+xml) and HTML (text/html). As a result, the filesize of the generated notebook can become large.

      In order to remedy this you can use the clever Julia package DisplayAs to limit the output capabilities of and object. For example, to "force" an image to be captures as image/png only, you can use

      import DisplayAs
 img = plot(...)
 img = DisplayAs.PNG(img)

      This can save some memory, since the image is never captured in e.g. SVG or HTML formats.

      Note

      It is best to always let the object be showable as text/plain. This can be achieved by nested calls to DisplayAs output types. For example, to limit an image img to be showable as just image/png and text/plain you can use

      img = plot(...)
-img = DisplayAs.Text(DisplayAs.PNG(img))
      +img = DisplayAs.Text(DisplayAs.PNG(img))