much more docs

README.md

@@ -4,10 +4,17 @@
 |:--------------------------------------- |:----------------------------------------------------------------------------------------------- |
 | [![][docs-latest-img]][docs-latest-url] | [![][travis-img]][travis-url] [![][appveyor-img]][appveyor-url] [![][codecov-img]][codecov-url] |
 
-`Literate.jl` is a package that, based on a single source file, generates markdown,
-for e.g. [Documenter.jl](https://github.com/JuliaDocs/Documenter.jl),
-[Jupyter notebooks](http://jupyter.org/) and uncommented scripts for documentation
-of your package.
+Literate is a package for [Literate Programming](https://en.wikipedia.org/wiki/Literate_programming).
+The main purpose is to facilitate writing Julia examples/tutorials that can be included in
+your package documentation.
+
+Literate can generate markdown pages
+(for e.g. [Documenter.jl](https://github.com/JuliaDocs/Documenter.jl)), and
+[Jupyter notebooks](http://jupyter.org/), from the same source file. There is also
+an option to "clean" the source from all metadata, and produce a pure Julia script.
+Using a single source file for multiple purposes reduces maintenance, and makes sure
+your different output formats are synced with each other.
+
 
 
 [docs-latest-img]: https://img.shields.io/badge/docs-latest-blue.svg

docs/make.jl

@@ -5,9 +5,15 @@ using Literate
 EXAMPLE = joinpath(@__DIR__, "..", "examples", "example.jl")
 OUTPUT = joinpath(@__DIR__, "src/generated")
 
-Literate.markdown(EXAMPLE, OUTPUT)
-Literate.notebook(EXAMPLE, OUTPUT)
-Literate.script(EXAMPLE, OUTPUT)
+function preprocess(str)
+    str = replace(str, "MYVARIABLE" => "z")
+    str = replace(str, "MYVALUE" => "1.0 + 2.0im")
+    return str
+end
+
+Literate.markdown(EXAMPLE, OUTPUT, preprocess = preprocess)
+Literate.notebook(EXAMPLE, OUTPUT, preprocess = preprocess)
+Literate.script(EXAMPLE, OUTPUT, preprocess = preprocess)
 
 makedocs(
     modules = [Literate],
@@ -26,6 +32,7 @@ makedocs(
 deploydocs(
     repo = "github.com/fredrikekre/Literate.jl.git",
     target = "build",
+    julia = "0.6",
     deps = nothing,
     make = nothing
 )

docs/src/customprocessing.md

@@ -4,8 +4,8 @@ Since all packages are different, and may have different demands on how
 to create a nice example for the documentation it is important that
 the package maintainer does not feel limited by the by default provided syntax
 that this package offers. While you can generally come a long way by utilizing
-[line filtering](@ref Filtering-lines) there might be situations where you need
-to manually hook into the generation and change things. In `Literate.jl` this
+[line filtering](@ref Filtering-Lines) there might be situations where you need
+to manually hook into the generation and change things. In Literate this
 is done by letting the user supply custom pre- and post-processing functions
 that may do transformation of the content.
 

docs/src/documenter.md

@@ -1,14 +1,14 @@
 # [**6.** Interaction with Documenter.jl](@id Interaction-with-Documenter)
 
-`Literate.jl` can be used for any purpose, it spits out regular markdown files,
+Literate can be used for any purpose, it spits out regular markdown files,
 and notebooks. Typically, though, these files will be used to render documentation
 for your package. The generators ([`Literate.markdown`](@ref), [`Literate.notebook`](@ref)
 and [`Literate.script`](@ref)) supports a keyword argument `documenter` that lets
-the generator perform some extra things, keeping in mind that the generated files will,
-eventually, be used with Documenter.jl. So lets take a look at what will happen
+the generator perform some extra things, keeping in mind that the source code have been
+written with Documenter.jl in mind. So lets take a look at what will happen
 if we set `documenter = true`:
 
-[`Literate.markdown`](@ref):
+### [`Literate.markdown`](@ref):
 - The default code fence will change from
   ````
   ```julia
@@ -30,7 +30,7 @@ if we set `documenter = true`:
   ```
   ````
 
-[`Literate.notebook`](@ref):
+### [`Literate.notebook`](@ref):
 - Documenter style `@ref`s and `@id` will be removed. This means that you can use
   `@ref` and `@id` in the source file without them leaking to the notebook.
 - Documenter style markdown math
@@ -45,3 +45,7 @@ if we set `documenter = true`:
   \int f dx
   \end{equation}
   ```
+
+### [`Literate.script`](@ref):
+- Documenter style `@ref`s and `@id` will be removed. This means that you can use
+  `@ref` and `@id` in the source file without them leaking to the script.

docs/src/fileformat.md

@@ -1,6 +1,6 @@
 # **2.** File Format
 
-The source file format for `Literate.jl` is a regular, commented, julia (`.jl`) scripts.
+The source file format for Literate is a regular, commented, julia (`.jl`) scripts.
 The idea is that the scripts also serve as documentation on their own and it is also
 simple to include them in the test-suite, with e.g. `include`, to make sure the examples
 stay up do date with other changes in your package.
@@ -30,61 +30,19 @@ In the lines `#'` we can use regular markdown syntax, for example the `#`
 used for the heading and the backticks for formatting code. The other lines are regular
 julia code. We note a couple of things:
 - The script is valid julia, which means that we can `include` it and the example will run
+  (for example in the `test/runtests.jl` script, to include the example in the test suite).
 - The script is "self-explanatory", i.e. the markdown lines works as comments and
   thus serve as good documentation on its own.
 
-For simple use this is all you need to know, the script above is valid. Let's take a look
-at what the above snippet would generate, with default settings:
+For simple use this is all you need to know. The following additional special syntax can also be used:
+- `#md`, `#nb`, `#jl`: tags for filtering of lines, see [Filtering Lines](@ref Filtering-Lines).
+- `#-`: tag for manually controlling chunk-splits, see [Custom control over chunk splits](@ref).
 
-- [`Literate.markdown`](@ref): leading `#'` are removed, and code lines are wrapped in
-  `@example`-blocks:
-  ````markdown
-  # Rational numbers
+There is also some default convenience replacements that will always be performed, see
+[Default Replacements](@ref).
 
-  In julia rational numbers can be constructed with the `//` operator.
-  Lets define two rational numbers, `x` and `y`:
 
-  ```@example filename
-  x = 1//3
-  y = 2//5
-  ```
-
-  When adding `x` and `y` together we obtain a new rational number:
-
-  ```@example filename
-  z = x + y
-  ```
-  ````
-
-- [`Literate.notebook`](@ref): leading `#'` are removed, markdown lines are placed in
-  `"markdown"` cells, and code lines in `"code"` cells:
-  ```
-           │ # Rational numbers
-           │
-           │ In julia rational numbers can be constructed with the `//` operator.
-           │ Lets define two rational numbers, `x` and `y`:
-
-  In [1]:  │ x = 1//3
-           │ y = 2//5
-
-  Out [1]: │ 2//5
-
-           │ When adding `x` and `y` together we obtain a new rational number:
-
-  In [2]:  │ z = x + y
-
-  Out [2]: │ 11//15
-  ```
-
-- [`Literate.script`](@ref): all lines starting with `#'` are removed:
-  ```julia
-  x = 1//3
-  y = 2//5
-
-  z = x + y
-  ```
-
-## [**2.2.** Filtering Lines](@id Filtering-lines)
+## [**2.2.** Filtering Lines](@id Filtering-Lines)
 
 It is possible to filter out lines depending on the output format. For this purpose,
 there are three different "tokens" that can be placed on the start of the line:
@@ -109,3 +67,29 @@ is a case where we can prepend `#md` to those lines:
 The lines in the example above would be filtered out in the preprocessing step, unless we are
 generating a markdown file. When generating a markdown file we would simple remove
 the leading `#md ` from the lines. Beware that the space after the tag is also removed.
+
+
+## [**2.3.** Default Replacements](@id Default-Replacements)
+
+The following convenience "macros" are always expanded:
+
+- `@__NAME__`
+
+  expands to the `name` keyword argument to [`Literate.markdown`](@ref),
+  [`Literate.notebook`](@ref) and [`Literate.script`](@ref)
+  (defaults to the filename of the input file).
+
+- `@__REPO__ROOT_URL__`
+
+  expands to `https://github.com/$(ENV["TRAVIS_REPO_SLUG"])/blob/master/`
+  and is a convenient way to use when you want to link to files outside the
+  doc-build directory. For example `@__REPO__ROOT_URL__src/Literate.jl` would link
+  to the source of the Literate module.
+
+- `@__NBVIEWER_ROOT_URL__`
+
+  expands to
+  `https://nbviewer.jupyter.org/github/$(ENV["TRAVIS_REPO_SLUG"])/blob/gh-pages/$(folder)/`
+  where `folder` is the folder that `Documenter.deploydocs` deploys too.
+  This can be used if you want a link that opens the generated notebook in
+  [http://nbviewer.jupyter.org/](http://nbviewer.jupyter.org/).

docs/src/index.md

@@ -1,51 +1,48 @@
 # **1.** Introduction
 
-Welcome to the documentation for `Literate.jl`. A simplistic package
-to help you organize examples for you package documentation.
+Welcome to the documentation for Literate -- a simplistic package
+for [Literate Programming](https://en.wikipedia.org/wiki/Literate_programming).
 
 ### What?
 
-`Literate.jl` is a package that, based on a single source file, generates markdown,
-for e.g. [Documenter.jl](https://github.com/JuliaDocs/Documenter.jl),
-[Jupyter notebooks](http://jupyter.org/) and uncommented scripts for documentation
-of your package.
+Literate is a package that generate markdown pages
+(for e.g. [Documenter.jl](https://github.com/JuliaDocs/Documenter.jl)), and
+[Jupyter notebooks](http://jupyter.org/), from the same source file. There is also
+an option to "clean" the source from all metadata, and produce a pure Julia script.
 
 The main design goal is simplicity. It should be simple to use, and the syntax should
-be simple. In short all you have to do is to write a commented julia script!
+be simple. In short, all you have to do is to write a commented julia script!
 
-The package consists mainly of three functions, which all takes the same script file
+The public interface consists mainly of three functions, which all takes the same script file
 as input, but generates different output:
 - [`Literate.markdown`](@ref): generates a markdown file
 - [`Literate.notebook`](@ref): generates an (optionally executed) notebook
-- [`Literate.script`](@ref): generates a plain script file, removing everything
-  that is not code
+- [`Literate.script`](@ref): generates a plain script file, removing all metadata
+  and special syntax.
 
 ### Why?
 
-Literate are (probably) the best way to showcase your awesome package, and examples
+Examples are (probably) the best way to showcase your awesome package, and examples
 are often the best way for a new user to learn how to use it. It is therefore important
 that the documentation of your package contains examples for users to read and study.
 However, people are different, and we all prefer different ways of trying out a new
 package. Some people wants to RTFM, others want to explore the package interactively in,
 for example, a notebook, and some people wants to study the source code. The aim of
-`Literate.jl` is to make it easy to give the user all of these options, while still
+Literate is to make it easy to give the user all of these options, while still
 keeping maintenance to a minimum.
 
 It is quite common that packages have "example notebooks" to showcase the package.
-Notebooks are great for this, but they are not so great with version control, like git.
-The reason is that a notebook is a very "rich" format since it contains output and other
-metadata. Changes to the notebook thus result in large diffs, which makes it harder to
-review the actual changes.
+Notebooks are great for showcasing a package, but they are not so great with version
+control, like git. The reason being that a notebook is a very "rich" format since it
+contains output and other metadata. Changes to the notebook thus result in large diffs,
+which makes it harder to review the actual changes.
 
 It is also common that packages include examples in the documentation, for example
 by using [Documenter.jl](https://github.com/JuliaDocs/Documenter.jl) `@example`-blocks.
 This is also great, but it is not quite as interactive as a notebook, for the users
 who prefer that.
 
-`Literate.jl` tries to solve the problems above by creating the output as a part of the doc
-build. `Literate.jl` generates the output from a single source file which makes it easier to
-maintain, test, and keep the manual and your example notebooks in sync.
+Literate tries to solve the problems above by creating the output as a part of the doc
+build. Literate generates the output based on a single source file which makes it
+easier to maintain, test, and keep the manual and your example notebooks in sync.
 
-### How?
-
-TBD

docs/src/outputformats.md

@@ -1,49 +1,107 @@
-# [**4.** Output formats](@id Output-formats)
+# [**4.** Output Formats](@id Output-Formats)
 
+When the source is parsed, and have been processed it is time to render the output.
+We will consider the following source snippet:
 
-## [**4.1.** Markdown output](@id Markdown-output)
+```julia
+#' # Rational numbers
+#'
+#' In julia rational numbers can be constructed with the `//` operator.
+#' Lets define two rational numbers, `x` and `y`:
 
-````
-#' # Markdown                                                  ┐
-#'                                                             │
-#' This line is treated as markdown, since it starts with #'   │
-#' The leading #' (including the space) is removed             ┘
+x = 1//3
+#-
+y = 2//5
 
-#' Here is an example with some code                           ]
+#' When adding `x` and `y` together we obtain a new rational number:
 
-x = sin.(cos.([1, 2, 3]))                                      ┐
-y = x.^2 - x                                                   ┘
-````
+z = x + y
+```
+
+and see how this is rendered in each of the output formats.
 
-By default, `CodeChunks` written to Documenter `@example` blocks. For example,
-the code above would result in the following markdown:
+## [**4.1.** Markdown Output](@id Markdown-Output)
+
+The (default) markdown output of the source snippet above is as follows
 
 ````markdown
-# Markdown
+# Rational numbers
 
-This line is treated as markdown, since it starts with #'
-The leading #' (including the space) is removed
+In julia rational numbers can be constructed with the `//` operator.
+Lets define two rational numbers, `x` and `y`:
 
-Here is an example with some code
+```@example name
+x = 1//3
+```
 
-```@example
-x = sin.(cos.([1, 2, 3]))
-y = x.^2 - x
+```@example name
+y = 2//5
+```
+
+When adding `x` and `y` together we obtain a new rational number:
+
+```@example name
+z = x + y
 ```
 ````
 
+We note that lines starting with `#'` is printed as regular markdown,
+and the code lines have been wrapped in `@example` blocks.
+
+Some of the output rendering can be controlled with keyword arguments to
+[`Literate.markdown`](@ref):
+
 ```@docs
 Literate.markdown
 ```
 
-## [**4.2.** Notebook output](@id Notebook-output)
+## [**4.2.** Notebook Output](@id Notebook-Output)
+
+The (default) notebook output of the source snippet above is as follows
+
+```
+        │ # Rational numbers
+        │
+        │ In julia rational numbers can be constructed with the `//` operator.
+        │ Lets define two rational numbers, `x` and `y`:
+
+In[1]:  │ x = 1//3
+Out[1]: │ 1//3
+
+In[2]:  │ y = 2//5
+Out[2]: │ 2//5
+
+        │ When adding `x` and `y` together we obtain a new rational number:
+
+In[3]:  │ z = x + y
+Out[3]: │ 11/15
+```
+
+We note that lines starting with `#'` is put in markdown cells,
+and the code lines have been put in code cells. By default the notebook
+is also executed and output cells populated. Some of the output rendering
+can be controlled with keyword arguments to [`Literate.notebook`](@ref):
 
 ```@docs
 Literate.notebook
 ```
 
 
-## [**4.3.** Script output](@id Script-output)
+## [**4.3.** Script Output](@id Script-Output)
+
+The (default) script output of the source snippet above is as follows
+
+```julia
+x = 1//3
+
+y = 2//5
+
+z = x + y
+```
+
+We note that lines starting with `#'` are removed and only the
+code lines have been kept. Some of the output rendering can be controlled
+with keyword arguments to [`Literate.script`](@ref):
 
 ```@docs
 Literate.script

docs/src/pipeline.md

@@ -10,14 +10,17 @@ The generation of output follows the same pipeline for all output formats:
 
 ## [**3.1.** Pre-processing](@id Pre-processing)
 
-The first step is pre-processing of the input file. The file is read to a `String`
-and CRLF style line endings (`"\r\n"`) are replaced with LF line endings (`"\n"`) to simplify
-internal processing. The next step is to apply the user specified pre-processing function.
-See [Custom pre- and post-processing](@ref Custom-pre-and-post-processing).
+The first step is pre-processing of the input file. The file is read to a `String`.
+The first processing step is to apply the user specified pre-processing function,
+see [Custom pre- and post-processing](@ref Custom-pre-and-post-processing).
 
-Next the line filtering is performed, see [Filtering lines](@ref), meaning that lines
-starting with `#md `, `#nb ` or `#jl ` are handled (either just the token itself is removed,
-or the full line, depending on the output target).
+The next step is to perform all of the built-in default replacements.
+CRLF style line endings (`"\r\n"`) are replaced with LF line endings (`"\n"`) to simplify
+internal processing. Next, line filtering is performed, see [Filtering Lines](@ref),
+meaning that lines starting with `#md `, `#nb ` or `#jl ` are handled (either just
+the token itself is removed, or the full line, depending on the output target).
+The last pre-processing step is to expand the convenience "macros" described
+in [Default Replacements](@ref) is expanded.
 
 
 ## [**3.2.** Parsing](@id Parsing)
@@ -108,8 +111,8 @@ The example above would result in two consecutive code-chunks.
 
 After the parsing it is time to generate the output. What is done in this step is
 very different depending on the output target, and it is describe in more detail in
-the Output format sections: [Markdown output](@ref), [Notebook output](@ref) and
-[Script output](@ref). In short, the following is happening:
+the Output format sections: [Markdown Output](@ref), [Notebook Output](@ref) and
+[Script Output](@ref). In short, the following is happening:
 
 * Markdown output: markdown chunks are printed as-is, code chunks are put inside
   a code fence (defaults to `@example`-blocks),

examples/example.jl

@@ -1,19 +1,42 @@
 #' # **7.** Example
 #'
-#' This is an example for Literate.jl.
-#' The source file can be found [here](@__REPO_ROOT_URL__examples/example.jl).
-#' The generated markdown can be found here: [`example.md`](./example.md), the
-#' generated notebook can be found here:
-#' [`example.ipynb`](@__NBVIEWER_ROOT_URL__generated/example.ipynb), and the
-#' plain script output can be found here: [`example.jl`](./example.jl).
+#' This is an example generated with Literate based on this
+#' source file: [`example.jl`](@__REPO_ROOT_URL__examples/example.jl).
+#' You are seeing the
+#md #' html-output which Documenter have generated based on a markdown
+#md #' file generated with Literate. The corresponding notebook
+#md #' can be found here: [`example.ipynb`](@__NBVIEWER_ROOT_URL__generated/example.ipynb),
+#nb #' generated notebook output. The corresponding markdown (html) output The generated markdown can be found here: [`example.md`](./example.md), the
+#nb #' can be found here: [`example.md`](./example.md),
+#' and the plain script output can be found here: [`example.jl`](./example.jl).
 
-#' ### Rational numbers in Julia
-#' Rational number in julia can be constructed with the `//` operator:
+#' It is recommended to have the [source file](@__REPO_ROOT_URL__examples/example.jl)
+#' available when reading this, to better understand how the syntax in the source file
+#' corresponds to the output you are seeing.
+
+#' ### Basic syntax
+#' The basic syntax for Literate is simple, lines starting with `#'` is interpreted
+#' as markdown, and all the other lines are interpreted as code. Here is some code:
 
 x = 1//3
 y = 2//5
 
-#' Operations with rational number returns a new rational number
+#' In markdown sections we can use markdown syntax. For example, we can
+#' write *text in italic font*, **text in bold font** and use
+#' [links](https://www.youtube.com/watch?v=dQw4w9WgXcQ).
+
+#' It is possible to filter lines by starting it with `#md`, `#nb` or `#jl`
+#' for markdown, notebook and script output only, respectively.
+#md #' This line is filtered out for notebook and script output.
+#nb #' This line is filtered out for markdown and script output.
+#jl #' This line is filtered out for markdown and notebook output.
+
+#' The source file is parsed in chunks of markdown and code. Starting a line
+#' with `#-` manually inserts a chunk break. For example, if we want to
+#' display the output of the following operations we may insert `#-` in
+#' between. These two code blocks will now end up in different
+#' `@example`-blocks in the markdown output, and two different notebook cells
+#' in the notebook output.
 
 x + y
 
@@ -21,11 +44,25 @@ x + y
 
 x * y
 
-#' Everytime a rational number is constructed, it will be simplified
-#' using the `gcd` function, for example `2//4` simplifies to `1//2`:
-
-2//4
+#' ### Custom processing
+#'
+#' It is possible to give Literate custom pre- and post-processing functions.
+#' For example, here we insert two placeholders, which we will replace with
+#' something else at time of generation. We have here replaced our placeholders
+#' with `z` and `1.0 + 2.0im`:
 
-#' and `2//4 + 2//4` simplifies to `1//1`:
+MYVARIABLE = MYVALUE
 
-2//4 + 2//4
+#' ### [Documenter.jl interaction](@id documenter-interaction)
+#'
+#' In the source file it is possible to use Documenter.jl style references,
+#' such as `@ref` and `@id`. These will be filtered out in the notebook output.
+#' For example, [here is a link](@ref documenter-interaction), but it is only
+#' visible as a link if you are reading the markdown output. We can also
+#' use equations:
+#' ```math
+#' \int f(x) dx
+#' ```
+#' using Documenters math syntax. This is automatically changed to
+#' `\begin{equation} ... \end{equation}` in the notebook output to display
+#' correctly in the notebook too.

src/Literate.jl

@@ -168,11 +168,14 @@ filename(str) = first(splitext(last(splitdir(str))))
 Generate a plain script file from `inputfile` and write the result to `outputdir`.
 
 Keyword arguments:
-- `name`: name of the output file, excluding `.jl`. Defaults to the
-  filename of `inputfile`.
+- `name`: name of the output file, excluding `.jl`. `name` is also used to
+  replace `@__NAME__`. Defaults to the filename of `inputfile`.
 - `preprocess`, `postprocess`: custom pre- and post-processing functions,
   see the [Custom pre- and post-processing](@ref Custom-pre-and-post-processing)
   section of the manual. Defaults to `identity`.
+- `documenter`: boolean that says if the source contains Documenter.jl specific things
+  to filter out during script generation. Defaults to `true`. See the the manual
+  section on [Interaction with Documenter](@ref Interaction-with-Documenter).
 """
 function script(inputfile, outputdir; preprocess = identity, postprocess = identity,
                 name = filename(inputfile), documenter = true, kwargs...)
@@ -223,7 +226,8 @@ to the directory`outputdir`.
 
 Keyword arguments:
 - `name`: name of the output file, excluding `.md`. `name` is also used to name
-  all the `@example` blocks. Defaults to the filename of `inputfile`.
+  all the `@example` blocks, and to replace `@__NAME__`.
+  Defaults to the filename of `inputfile`.
 - `preprocess`, `postprocess`: custom pre- and post-processing functions,
   see the [Custom pre- and post-processing](@ref Custom-pre-and-post-processing)
   section of the manual. Defaults to `identity`.
@@ -316,8 +320,8 @@ const JUPYTER_VERSION = v"4.3.0"
 Generate a notebook from `inputfile` and write the result to `outputdir`.
 
 Keyword arguments:
-- `name`: name of the output file, excluding `.ipynb`. Defaults to the
-  filename of `inputfile`.
+- `name`: name of the output file, excluding `.ipynb`. `name` is also used to
+  replace `@__NAME__`. Defaults to the filename of `inputfile`.
 - `preprocess`, `postprocess`: custom pre- and post-processing functions,
   see the [Custom pre- and post-processing](@ref Custom-pre-and-post-processing)
   section of the manual. Defaults to `identity`.