From cd2e31193ce527e408ebb9389f36d7403629f9e6 Mon Sep 17 00:00:00 2001 From: autodocs Date: Tue, 9 Oct 2018 11:26:54 +0000 Subject: [PATCH] build based on 142c511 --- dev/customprocessing.html | 25 ++++- dev/generated/example.html | 84 ++++++++--------- dev/generated/example.ipynb | 172 +++++++++++++++++------------------ dev/generated/notebook.ipynb | 4 +- dev/outputformats.html | 4 +- dev/search_index.js | 18 +++- 6 files changed, 172 insertions(+), 135 deletions(-) diff --git a/dev/customprocessing.html b/dev/customprocessing.html index 749ceca..9253605 100644 --- a/dev/customprocessing.html +++ b/dev/customprocessing.html @@ -1,8 +1,29 @@ -5. Custom pre- and post-processing · Literate.jl
5. Custom pre- and post-processing

5. Custom pre- and post-processing

Since all packages are different, and may have different demands on how to create a nice example for the documentation it is important that the package maintainer does not feel limited by the by default provided syntax that this package offers. While you can generally come a long way by utilizing line filtering there might be situations where you need to manually hook into the generation and change things. In Literate this is done by letting the user supply custom pre- and post-processing functions that may do transformation of the content.

All of the generators (Literate.markdown, Literate.notebook and Literate.script) accepts preprocess and postprocess keyword arguments. The default "transformation" is the identity function. The input to the transformation functions is a String, and the output should be the transformed String.

preprocess is sent the raw input that is read from the source file (modulo the default line ending transformation). postprocess is given different things depending on the output: For markdown and script output postprocess is given the content String just before writing it to the output file, but for notebook output postprocess is given the dictionary representing the notebook, since, in general, this is more useful.

As an example, lets say we want to splice the date of generation into the output. We could of course update our source file before generating the docs, but we could instead use a preprocess function that splices the date into the source for us. Consider the following source file:

# # Example
+5. Custom pre- and post-processing · Literate.jl
5. Custom pre- and post-processing
5. Custom pre- and post-processing
Since all packages are different, and may have different demands on how to create a nice example for the documentation it is important that the package maintainer does not feel limited by the by default provided syntax that this package offers. While you can generally come a long way by utilizing line filtering there might be situations where you need to manually hook into the generation and change things. In Literate this is done by letting the user supply custom pre- and post-processing functions that may do transformation of the content.
All of the generators (Literate.markdown, Literate.notebook and Literate.script) accepts preprocess and postprocess keyword arguments. The default "transformation" is the identity function. The input to the transformation functions is a String, and the output should be the transformed String.
preprocess is sent the raw input that is read from the source file (modulo the default line ending transformation). postprocess is given different things depending on the output: For markdown and script output postprocess is given the content String just before writing it to the output file, but for notebook output postprocess is given the dictionary representing the notebook, since, in general, this is more useful.
Example: Adding current date
As an example, lets say we want to splice the date of generation into the output. We could of course update our source file before generating the docs, but we could instead use a preprocess function that splices the date into the source for us. Consider the following source file:
# # Example
 # This example was generated DATEOFTODAY
 
 x = 1 // 3
where DATEOFTODAY is a placeholder, to make it easier for our preprocess function to find the location. Now, lets define the preprocess function, for example
function update_date(content)
     content = replace(content, "DATEOFTODAY" => Date(now()))
     return content
-end
which would replace every occurrence of "DATEOFTODAY" with the current date. We would now simply give this function to the generator, for example:
Literate.markdown("input.jl", "outputdir"; preprocess = update_date)

+end

which would replace every occurrence of "DATEOFTODAY" with the current date. We would now simply give this function to the generator, for example:

Literate.markdown("input.jl", "outputdir"; preprocess = update_date)

Example: Replacing include calls with included code

Let's say that we have some individual example files file1, file2, ... etc. that are runnable and also following the style of Literate. These files could be for example used in the test suite of your package.

We want to group them all into a single page in our documentation, but we do not want to copy paste the content of file1, ... for robustness: the files are included in the test suite and some changes may occur to them. We want these changes to also be reflected in the documentation.

A very easy way to do this is using preprocess to interchange include statements with file content. First, create a runnable .jl following the format of Literate

# # Replace includes
+# This is an example to replace `include` calls with the actual file content.
+
+include("file1.jl")
+
+# Cool, we just saw the result of the above code snippet. Here is one more:
+
+include("file2.jl")

Let's say we have saved this file as examples.jl. Then, you want to properly define a pre-processing function:

function replace_includes(str)
+
+    included = ["file1.jl", "file2.jl"]
+
+    # Here the path loads the files from their proper directory,
+    # which may not be the directory of the `examples.jl` file!
+    path = "directory/to/example/files/"
+
+    for ex in included
+        content = read(path*ex, String)
+        str = replace(str, "include(\"$(ex)\")" => content)
+    end
+    return str
+end

(of course replace included with your respective files)

Finally, you simply pass this function to e.g. Literate.markdown as

Literate.markdown("examples.jl", "path/to/save/markdown";
+                  name = "markdown_file_name", preprocess = replace_includes)

and you will see that in the final output file (here markdown_file_name.md) the include statements are replaced with the actual code to be included!

This approach is used for example in the documentation of the Julia package TimeseriesPrediction, see here and here for the generating script

diff --git a/dev/generated/example.html b/dev/generated/example.html index 121b334..7300886 100644 --- a/dev/generated/example.html +++ b/dev/generated/example.html @@ -17,114 +17,114 @@ y2 = cos.(x) plot(x, [y1, y2]) - + - - + - - + - - - - - - - - - - - - - - - - - - - - - + 0 - + 5 - + 10 - + 15 - + -1.0 - + -0.5 - + 0.0 - + 0.5 - + 1.0 - 559.828,227.723 560.354,224.45 560.88,221.169 561.405,217.879 561.931,214.582 562.457,211.279 562.982,207.972 563.508,204.661 564.034,201.347 564.56,198.032 "/> - 559.828,24.8788 560.354,24.3495 560.88,23.8819 561.405,23.4764 561.931,23.133 562.457,22.8519 562.982,22.6332 563.508,22.4769 564.034,22.3831 564.56,22.3518 "/> - - - - + y1 - - + y2 diff --git a/dev/generated/example.ipynb b/dev/generated/example.ipynb index 64dff57..cf50d2d 100644 --- a/dev/generated/example.ipynb +++ b/dev/generated/example.ipynb @@ -187,114 +187,114 @@ "\n", "\n", "\n", - " \n", + " \n", " \n", " \n", "\n", - "\n", "\n", - " \n", + " \n", " \n", " \n", "\n", - "\n", "\n", - " \n", + " \n", " \n", " \n", "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", + "\n", "0\n", "\n", - "\n", + "\n", "5\n", "\n", - "\n", + "\n", "10\n", "\n", - "\n", + "\n", "15\n", "\n", - "\n", + "\n", "-1.0\n", "\n", - "\n", + "\n", "-0.5\n", "\n", - "\n", + "\n", "0.0\n", "\n", - "\n", + "\n", "0.5\n", "\n", - "\n", + "\n", "1.0\n", "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", + "\n", "y1\n", "\n", - "\n", - "\n", + "\n", "y2\n", "\n", "\n" @@ -524,114 +524,114 @@ "\n", "\n", "\n", - " \n", + " \n", " \n", " \n", "\n", - "\n", "\n", - " \n", + " \n", " \n", " \n", "\n", - "\n", "\n", - " \n", + " \n", " \n", " \n", "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", + "\n", "0\n", "\n", - "\n", + "\n", "5\n", "\n", - "\n", + "\n", "10\n", "\n", - "\n", + "\n", "15\n", "\n", - "\n", + "\n", "-1.0\n", "\n", - "\n", + "\n", "-0.5\n", "\n", - "\n", + "\n", "0.0\n", "\n", - "\n", + "\n", "0.5\n", "\n", - "\n", + "\n", "1.0\n", "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", + "\n", "y1\n", "\n", - "\n", - "\n", + "\n", "y2\n", "\n", "\n" @@ -940,11 +940,11 @@ "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", - "version": "1.0.0" + "version": "1.0.1" }, "kernelspec": { "name": "julia-1.0", - "display_name": "Julia 1.0.0", + "display_name": "Julia 1.0.1", "language": "julia" } }, diff --git a/dev/generated/notebook.ipynb b/dev/generated/notebook.ipynb index 44380f6..fe325ea 100644 --- a/dev/generated/notebook.ipynb +++ b/dev/generated/notebook.ipynb @@ -88,11 +88,11 @@ "file_extension": ".jl", "mimetype": "application/julia", "name": "julia", - "version": "1.0.0" + "version": "1.0.1" }, "kernelspec": { "name": "julia-1.0", - "display_name": "Julia 1.0.0", + "display_name": "Julia 1.0.1", "language": "julia" } }, diff --git a/dev/outputformats.html b/dev/outputformats.html index 07f2788..6b3555d 100644 --- a/dev/outputformats.html +++ b/dev/outputformats.html @@ -31,8 +31,8 @@ When adding `x` and `y` together we obtain a new rational number: ```@example name z = x + y -```

We note that lines starting with # are printed as regular markdown, and the code lines have been wrapped in @example blocks. We also note that an @meta block have been added, that sets the EditURL variable. This is used by Documenter to redirect the "Edit on GitHub" link for the page, see Interaction with Documenter.

Some of the output rendering can be controlled with keyword arguments to Literate.markdown:

Literate.markdown — Function.
Literate.markdown(inputfile, outputdir; kwargs...)

Generate a markdown file from inputfile and write the result to the directoryoutputdir.

Keyword arguments:

  • name: name of the output file, excluding .md. name is also used to name all the @example blocks, and to replace @__NAME__. Defaults to the filename of inputfile.
  • preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.
  • documenter: boolean that tells if the output is intended to use with Documenter.jl. Defaults to true. See the the manual section on Interaction with Documenter.
  • codefence: A Pair of opening and closing code fence. Defaults to
    "```@example $(name)" => "```"
    if documenter = true and
    "```julia" => "```"
    if documenter = false.
  • credit: boolean that controls the addition of This file was generated with Literate.jl ... to the bottom of the page. If you find Literate.jl useful then feel free to keep this to the default, which is true.
source

4.2. Notebook Output

The (default) notebook output of the source snippet can be seen here: notebook.ipynb.

We note that lines starting with # are placed in markdown cells, and the code lines have been placed in code cells. By default the notebook is also executed and output cells populated. The current working directory is set to the specified output directory the notebook is executed. Some of the output rendering can be controlled with keyword arguments to Literate.notebook:

Literate.notebook — Function.
Literate.notebook(inputfile, outputdir; kwargs...)

Generate a notebook from inputfile and write the result to outputdir.

Keyword arguments:

  • name: name of the output file, excluding .ipynb. name is also used to replace @__NAME__. Defaults to the filename of inputfile.
  • preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.
  • execute: a boolean deciding if the generated notebook should also be executed or not. Defaults to true. The current working directory is set to outputdir when executing the notebook.
  • documenter: boolean that says if the source contains Documenter.jl specific things to filter out during notebook generation. Defaults to true. See the the manual section on Interaction with Documenter.
  • credit: boolean that controls the addition of This file was generated with Literate.jl ... to the bottom of the page. If you find Literate.jl useful then feel free to keep this to the default, which is true.
source

4.3. Script Output

The (default) script output of the source snippet above is as follows

x = 1//3
+```

We note that lines starting with # are printed as regular markdown, and the code lines have been wrapped in @example blocks. We also note that an @meta block have been added, that sets the EditURL variable. This is used by Documenter to redirect the "Edit on GitHub" link for the page, see Interaction with Documenter.

Some of the output rendering can be controlled with keyword arguments to Literate.markdown:

Literate.markdown — Function.
Literate.markdown(inputfile, outputdir; kwargs...)

Generate a markdown file from inputfile and write the result to the directoryoutputdir.

Keyword arguments:

  • name: name of the output file, excluding .md. name is also used to name all the @example blocks, and to replace @__NAME__. Defaults to the filename of inputfile.
  • preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.
  • documenter: boolean that tells if the output is intended to use with Documenter.jl. Defaults to true. See the the manual section on Interaction with Documenter.
  • codefence: A Pair of opening and closing code fence. Defaults to
    "```@example $(name)" => "```"
    if documenter = true and
    "```julia" => "```"
    if documenter = false.
  • credit: boolean that controls the addition of This file was generated with Literate.jl ... to the bottom of the page. If you find Literate.jl useful then feel free to keep this to the default, which is true.
source

4.2. Notebook Output

The (default) notebook output of the source snippet can be seen here: notebook.ipynb.

We note that lines starting with # are placed in markdown cells, and the code lines have been placed in code cells. By default the notebook is also executed and output cells populated. The current working directory is set to the specified output directory the notebook is executed. Some of the output rendering can be controlled with keyword arguments to Literate.notebook:

Literate.notebook — Function.
Literate.notebook(inputfile, outputdir; kwargs...)

Generate a notebook from inputfile and write the result to outputdir.

Keyword arguments:

  • name: name of the output file, excluding .ipynb. name is also used to replace @__NAME__. Defaults to the filename of inputfile.
  • preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.
  • execute: a boolean deciding if the generated notebook should also be executed or not. Defaults to true. The current working directory is set to outputdir when executing the notebook.
  • documenter: boolean that says if the source contains Documenter.jl specific things to filter out during notebook generation. Defaults to true. See the the manual section on Interaction with Documenter.
  • credit: boolean that controls the addition of This file was generated with Literate.jl ... to the bottom of the page. If you find Literate.jl useful then feel free to keep this to the default, which is true.
source

4.3. Script Output

The (default) script output of the source snippet above is as follows

x = 1//3
 
 y = 2//5
 
-z = x + y

We note that lines starting with # are removed and only the code lines have been kept. Some of the output rendering can be controlled with keyword arguments to Literate.script:

Literate.script — Function.
Literate.script(inputfile, outputdir; kwargs...)

Generate a plain script file from inputfile and write the result to outputdir.

Keyword arguments:

  • name: name of the output file, excluding .jl. name is also used to replace @__NAME__. Defaults to the filename of inputfile.
  • preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.
  • documenter: boolean that says if the source contains Documenter.jl specific things to filter out during script generation. Defaults to true. See the the manual section on Interaction with Documenter.
  • keep_comments: boolean that, if set to true, keeps markdown lines as comments in the output script. Defaults to false.
  • credit: boolean that controls the addition of This file was generated with Literate.jl ... to the bottom of the page. If you find Literate.jl useful then feel free to keep this to the default, which is true.
source
+z = x + y

We note that lines starting with # are removed and only the code lines have been kept. Some of the output rendering can be controlled with keyword arguments to Literate.script:

Literate.script — Function.
Literate.script(inputfile, outputdir; kwargs...)

Generate a plain script file from inputfile and write the result to outputdir.

Keyword arguments:

  • name: name of the output file, excluding .jl. name is also used to replace @__NAME__. Defaults to the filename of inputfile.
  • preprocess, postprocess: custom pre- and post-processing functions, see the Custom pre- and post-processing section of the manual. Defaults to identity.
  • documenter: boolean that says if the source contains Documenter.jl specific things to filter out during script generation. Defaults to true. See the the manual section on Interaction with Documenter.
  • keep_comments: boolean that, if set to true, keeps markdown lines as comments in the output script. Defaults to false.
  • credit: boolean that controls the addition of This file was generated with Literate.jl ... to the bottom of the page. If you find Literate.jl useful then feel free to keep this to the default, which is true.
source
diff --git a/dev/search_index.js b/dev/search_index.js index 82a62c0..755933d 100644 --- a/dev/search_index.js +++ b/dev/search_index.js @@ -213,7 +213,23 @@ var documenterSearchIndex = {"docs": [ "page": "5. Custom pre- and post-processing", "title": "5. Custom pre- and post-processing", "category": "section", - "text": "Since all packages are different, and may have different demands on how to create a nice example for the documentation it is important that the package maintainer does not feel limited by the by default provided syntax that this package offers. While you can generally come a long way by utilizing line filtering there might be situations where you need to manually hook into the generation and change things. In Literate this is done by letting the user supply custom pre- and post-processing functions that may do transformation of the content.All of the generators (Literate.markdown, Literate.notebook and Literate.script) accepts preprocess and postprocess keyword arguments. The default \"transformation\" is the identity function. The input to the transformation functions is a String, and the output should be the transformed String.preprocess is sent the raw input that is read from the source file (modulo the default line ending transformation). postprocess is given different things depending on the output: For markdown and script output postprocess is given the content String just before writing it to the output file, but for notebook output postprocess is given the dictionary representing the notebook, since, in general, this is more useful.As an example, lets say we want to splice the date of generation into the output. We could of course update our source file before generating the docs, but we could instead use a preprocess function that splices the date into the source for us. Consider the following source file:# # Example\n# This example was generated DATEOFTODAY\n\nx = 1 // 3where DATEOFTODAY is a placeholder, to make it easier for our preprocess function to find the location. Now, lets define the preprocess function, for examplefunction update_date(content)\n content = replace(content, \"DATEOFTODAY\" => Date(now()))\n return content\nendwhich would replace every occurrence of \"DATEOFTODAY\" with the current date. We would now simply give this function to the generator, for example:Literate.markdown(\"input.jl\", \"outputdir\"; preprocess = update_date)" + "text": "Since all packages are different, and may have different demands on how to create a nice example for the documentation it is important that the package maintainer does not feel limited by the by default provided syntax that this package offers. While you can generally come a long way by utilizing line filtering there might be situations where you need to manually hook into the generation and change things. In Literate this is done by letting the user supply custom pre- and post-processing functions that may do transformation of the content.All of the generators (Literate.markdown, Literate.notebook and Literate.script) accepts preprocess and postprocess keyword arguments. The default \"transformation\" is the identity function. The input to the transformation functions is a String, and the output should be the transformed String.preprocess is sent the raw input that is read from the source file (modulo the default line ending transformation). postprocess is given different things depending on the output: For markdown and script output postprocess is given the content String just before writing it to the output file, but for notebook output postprocess is given the dictionary representing the notebook, since, in general, this is more useful." +}, + +{ + "location": "customprocessing.html#Example:-Adding-current-date-1", + "page": "5. Custom pre- and post-processing", + "title": "Example: Adding current date", + "category": "section", + "text": "As an example, lets say we want to splice the date of generation into the output. We could of course update our source file before generating the docs, but we could instead use a preprocess function that splices the date into the source for us. Consider the following source file:# # Example\n# This example was generated DATEOFTODAY\n\nx = 1 // 3where DATEOFTODAY is a placeholder, to make it easier for our preprocess function to find the location. Now, lets define the preprocess function, for examplefunction update_date(content)\n content = replace(content, \"DATEOFTODAY\" => Date(now()))\n return content\nendwhich would replace every occurrence of \"DATEOFTODAY\" with the current date. We would now simply give this function to the generator, for example:Literate.markdown(\"input.jl\", \"outputdir\"; preprocess = update_date)" +}, + +{ + "location": "customprocessing.html#Example:-Replacing-include-calls-with-included-code-1", + "page": "5. Custom pre- and post-processing", + "title": "Example: Replacing include calls with included code", + "category": "section", + "text": "Let\'s say that we have some individual example files file1, file2, ... etc. that are runnable and also following the style of Literate. These files could be for example used in the test suite of your package.We want to group them all into a single page in our documentation, but we do not want to copy paste the content of file1, ... for robustness: the files are included in the test suite and some changes may occur to them. We want these changes to also be reflected in the documentation.A very easy way to do this is using preprocess to interchange include statements with file content. First, create a runnable .jl following the format of Literate# # Replace includes\n# This is an example to replace `include` calls with the actual file content.\n\ninclude(\"file1.jl\")\n\n# Cool, we just saw the result of the above code snippet. Here is one more:\n\ninclude(\"file2.jl\")Let\'s say we have saved this file as examples.jl. Then, you want to properly define a pre-processing function:function replace_includes(str)\n\n included = [\"file1.jl\", \"file2.jl\"]\n\n # Here the path loads the files from their proper directory,\n # which may not be the directory of the `examples.jl` file!\n path = \"directory/to/example/files/\"\n\n for ex in included\n content = read(path*ex, String)\n str = replace(str, \"include(\\\"$(ex)\\\")\" => content)\n end\n return str\nend(of course replace included with your respective files)Finally, you simply pass this function to e.g. Literate.markdown asLiterate.markdown(\"examples.jl\", \"path/to/save/markdown\";\n name = \"markdown_file_name\", preprocess = replace_includes)and you will see that in the final output file (here markdown_file_name.md) the include statements are replaced with the actual code to be included!This approach is used for example in the documentation of the Julia package TimeseriesPrediction, see here and here for the generating script" }, {