extension preprocessor

Hi list,

I'm trying to use the extension preprocessor to transform the lines in an asciidoc file prior to processing.  Although I have something working, I find 'include::' macros are not getting processed, and am not sure how to proceed - I'm perhaps not understanding the rendering process correctly.

I'm currently using the 'process' method in a subclass of preprocessor, and changing or inserting lines in the 'lines' array, as appropriate.  I then return a new reader with these new lines.

    def process reader, lines
      return reader if lines.empty?

          # change and insert new rows in 'lines'

      return Asciidoctor::Reader.new lines
    end

This works well with a self-contained asciidoc file, but if the file contains 'include::' macros, these macros are not processed, and their text is simply inserted in the output html without rendering the document to insert.  I have tried 'image::' macros, and these are processed and the image added correctly.

Is the preprocessor an appropriate way to do a source-to-source transform?  I was hoping to slot my rewrite of the asciidoc document in as a first stage, aiming to return a set of lines in correct asciidoc syntax for later rendering.

The code I'm working on is here.

  thanks for any help,

       Peter.

---

If you reply to this email, your message will be added to the discussion below:

http://discuss.asciidoctor.org/extension-preprocessor-tp886.html

To start a new topic under Asciidoctor :: Discussion, email [hidden email]
To unsubscribe from Asciidoctor :: Discussion, click here.
NAML

Welcome Peter! We're glad to have you!

I'm thrilled to hear that you're working on an integration between Asciidoctor and asciidoc-bib. I've been looking forward to these two projects coming together. I was hopeful the extensions API would open that door, and it seems it has.

The preprocessor is the trickiest extension point to understand at first because it has the ability to affect the reader's cursor. Nothing some documentation can't solve :) So let's get to it!

You have two options when reading the lines in a preprocessor. You can either work with the raw lines of the main source document, or you can instruct the reader to walk the lines.

The second argument to the process method is the raw lines (original source) of the main document. Working with these lines is the most benign since it doesn't affect the reader's cursor. However, as you observed, preprocessor directives in those lines are not processed.

When you read the lines from the reader, any preprocessor directives, such as the include directive, will get processed. It will also advance the cursor, effectively discarding those lines. As such, you need to restore the lines by pushing them back onto the reader. You can so using the Reader#push_include method.

If the process method returns nil, Asciidoctor will continue processing using the reader in its current state. If you return a reader, it will replace the reader Asciidoctor uses.

Here's an example that shows the difference between the raw lines and lines read from the reader.

[source,ruby]
--

class SamplePreprocessor < Asciidoctor::Extensions::Preprocessor
  def process reader, raw_lines
    puts '.Raw lines'
    puts '....'
    puts raw_lines.join # <1>
    puts '....'
    puts "\n"
    lines = []
    while reader.has_more_lines?
      lines << reader.read_line # <2>
    end
    puts '.Preprocessed lines'
    puts '....'
    puts lines.join # <3>
    puts '....'
    use_push_include = true
    if use_push_include
      reader.push_include lines, '<stdin>', '<stdin>' # <4>
      nil # <5>
    else
      Asciidoctor::Reader.new lines # <6>
    end
  end
end
--

<1> Prints the original source of the main document

<2> Consume each line from the reader. The line read is the next line once preprocessor directives on the original line have been evaluated.

<3> Prints the effective lines once all preprocessor directives have been evaluated.

<4> Push the lines back onto the reader so that they can be interpreted by the AsciiDoc processor.
The second and third arguments (file and path, respectively) are required at the moment to work around a bug.

<5> A nil return instructs Asciidoctor to continue with the original Reader

<6> Replace the original Reader with a new one that contains the preprocessed lines

The only major caveat right now with reading the lines from the Reader is that it messes up the line information for reporting warnings. When you push the preprocessed lines back onto the Reader, Asciidoctor thinks they were all in the original file. I'm working on a way to retain this information in the next version.

If you are only reading information from the processed lines, and not modifying them, it's possible to push the raw lines back onto the reader, thus restoring its original state.

[source,ruby]
--

def process reader, raw_lines

  lines = reader.read # <1>

  cursor = reader.cursor
  reader.push_include raw_lines, (cursor.file || '<stdin>'), (cursor.path || '<stdin>'), 1 # <2>
  reader.process_lines = true if cursor.file.nil? # <3>
  nil

end
--

<1> Preprocess all lines and return as an Array

<2> Restore the original lines onto the Reader

<3> Work around an assumption that lines should not be processed if the file name is nil

I hope that clears some things up. Obviously, the Preprocessor extension needs some polishing, but that's what this discussion is helping to identify.

Cheers,

-Dan

On Sun, Oct 27, 2013 at 6:06 AM, plane [via Asciidoctor :: Discussion] <[hidden email]> wrote:

Hi list,

I'm trying to use the extension preprocessor to transform the lines in an asciidoc file prior to processing.  Although I have something working, I find 'include::' macros are not getting processed, and am not sure how to proceed - I'm perhaps not understanding the rendering process correctly.

I'm currently using the 'process' method in a subclass of preprocessor, and changing or inserting lines in the 'lines' array, as appropriate.  I then return a new reader with these new lines.

    def process reader, lines
      return reader if lines.empty?

          # change and insert new rows in 'lines'

      return Asciidoctor::Reader.new lines
    end

This works well with a self-contained asciidoc file, but if the file contains 'include::' macros, these macros are not processed, and their text is simply inserted in the output html without rendering the document to insert.  I have tried 'image::' macros, and these are processed and the image added correctly.

Is the preprocessor an appropriate way to do a source-to-source transform?  I was hoping to slot my rewrite of the asciidoc document in as a first stage, aiming to return a set of lines in correct asciidoc syntax for later rendering.

The code I'm working on is here.

  thanks for any help,

       Peter.

---

If you reply to this email, your message will be added to the discussion below:

http://discuss.asciidoctor.org/extension-preprocessor-tp886.html

To start a new topic under Asciidoctor :: Discussion, email [hidden email]
To unsubscribe from Asciidoctor :: Discussion, click here.
NAML

--

Dan Allen | http://google.com/profiles/dan.j.allen

[source,ruby]

--
Asciidoctor::Extensions.register do |document|
  preprocessor AsciidocBibExtension('biblio.bib')
end
--

You could assign the file name to a document attribute. That way, you could allow it to be overridden. For example:

[source,ruby]
--

Asciidoctor::Extensions.register do |document|

  unless document.attr? 'bibfile'

    document.set_attribute 'bibfile', 'biblio.bib' # <1>

  end
  preprocessor AsciidocBibExtension

end

--

<1> Set the bibfile attribute on the document.

The document may still override this value, unless the attribute was assigned via the CLI or API

-Dan

On Sun, Oct 27, 2013 at 3:53 PM, Dan Allen <[hidden email]> wrote:

Welcome Peter! We're glad to have you!

I'm thrilled to hear that you're working on an integration between Asciidoctor and asciidoc-bib. I've been looking forward to these two projects coming together. I was hopeful the extensions API would open that door, and it seems it has.

The preprocessor is the trickiest extension point to understand at first because it has the ability to affect the reader's cursor. Nothing some documentation can't solve :) So let's get to it!

You have two options when reading the lines in a preprocessor. You can either work with the raw lines of the main source document, or you can instruct the reader to walk the lines.

The second argument to the process method is the raw lines (original source) of the main document. Working with these lines is the most benign since it doesn't affect the reader's cursor. However, as you observed, preprocessor directives in those lines are not processed.

When you read the lines from the reader, any preprocessor directives, such as the include directive, will get processed. It will also advance the cursor, effectively discarding those lines. As such, you need to restore the lines by pushing them back onto the reader. You can so using the Reader#push_include method.

If the process method returns nil, Asciidoctor will continue processing using the reader in its current state. If you return a reader, it will replace the reader Asciidoctor uses.

Here's an example that shows the difference between the raw lines and lines read from the reader.

[source,ruby]
--

class SamplePreprocessor < Asciidoctor::Extensions::Preprocessor
  def process reader, raw_lines
    puts '.Raw lines'
    puts '....'
    puts raw_lines.join # <1>
    puts '....'
    puts "\n"
    lines = []
    while reader.has_more_lines?
      lines << reader.read_line # <2>
    end
    puts '.Preprocessed lines'
    puts '....'
    puts lines.join # <3>
    puts '....'
    use_push_include = true
    if use_push_include
      reader.push_include lines, '<stdin>', '<stdin>' # <4>
      nil # <5>
    else
      Asciidoctor::Reader.new lines # <6>
    end
  end
end
--

<1> Prints the original source of the main document

<2> Consume each line from the reader. The line read is the next line once preprocessor directives on the original line have been evaluated.

<3> Prints the effective lines once all preprocessor directives have been evaluated.

<4> Push the lines back onto the reader so that they can be interpreted by the AsciiDoc processor.
The second and third arguments (file and path, respectively) are required at the moment to work around a bug.

<5> A nil return instructs Asciidoctor to continue with the original Reader

<6> Replace the original Reader with a new one that contains the preprocessed lines

The only major caveat right now with reading the lines from the Reader is that it messes up the line information for reporting warnings. When you push the preprocessed lines back onto the Reader, Asciidoctor thinks they were all in the original file. I'm working on a way to retain this information in the next version.

If you are only reading information from the processed lines, and not modifying them, it's possible to push the raw lines back onto the reader, thus restoring its original state.

[source,ruby]
--

def process reader, raw_lines

  lines = reader.read # <1>

  cursor = reader.cursor
  reader.push_include raw_lines, (cursor.file || '<stdin>'), (cursor.path || '<stdin>'), 1 # <2>
  reader.process_lines = true if cursor.file.nil? # <3>
  nil

end
--

<1> Preprocess all lines and return as an Array

<2> Restore the original lines onto the Reader

<3> Work around an assumption that lines should not be processed if the file name is nil

I hope that clears some things up. Obviously, the Preprocessor extension needs some polishing, but that's what this discussion is helping to identify.

Cheers,

-Dan

On Sun, Oct 27, 2013 at 6:06 AM, plane [via Asciidoctor :: Discussion] <[hidden email]> wrote:

Hi list,

I'm trying to use the extension preprocessor to transform the lines in an asciidoc file prior to processing.  Although I have something working, I find 'include::' macros are not getting processed, and am not sure how to proceed - I'm perhaps not understanding the rendering process correctly.

I'm currently using the 'process' method in a subclass of preprocessor, and changing or inserting lines in the 'lines' array, as appropriate.  I then return a new reader with these new lines.

    def process reader, lines
      return reader if lines.empty?

          # change and insert new rows in 'lines'

      return Asciidoctor::Reader.new lines
    end

This works well with a self-contained asciidoc file, but if the file contains 'include::' macros, these macros are not processed, and their text is simply inserted in the output html without rendering the document to insert.  I have tried 'image::' macros, and these are processed and the image added correctly.

Is the preprocessor an appropriate way to do a source-to-source transform?  I was hoping to slot my rewrite of the asciidoc document in as a first stage, aiming to return a set of lines in correct asciidoc syntax for later rendering.

The code I'm working on is here.

  thanks for any help,

       Peter.

---

If you reply to this email, your message will be added to the discussion below:

http://discuss.asciidoctor.org/extension-preprocessor-tp886.html

To start a new topic under Asciidoctor :: Discussion, email [hidden email]
To unsubscribe from Asciidoctor :: Discussion, click here.
NAML

--

Dan Allen | http://google.com/profiles/dan.j.allen

--

Dan Allen | http://google.com/profiles/dan.j.allen

# TODO: Manage other asciidoctor options?
Asciidoctor.render_file ARGV[0], :safe => :safe, :in_place => true

You may want to delegate to the CLI invoker, essentially creating a cli wrapper:

[source,ruby]
--
require 'asciidoctor'
require 'asciidoc-bib' # includes and registers Asciidoctor extension
require 'asciidoctor/cli/options'
require 'asciidoctor/cli/invoker'

invoker = Asciidoctor::Cli::Invoker.new(ARGV)
invoker.invoke!
exit invoker.code
--

There's an issue [1] to add an argument to the commandline to require additional libraries, such as:

 $ asciidoctor -r asciidoc-bib sample.adoc

If you have other ideas, please feel free to suggest them in that issue.

-Dan

[1] https://github.com/asciidoctor/asciidoctor/issues/574

On Sun, Oct 27, 2013 at 4:03 PM, Dan Allen <[hidden email]> wrote:

[source,ruby]

--
Asciidoctor::Extensions.register do |document|
  preprocessor AsciidocBibExtension('biblio.bib')
end
--

You could assign the file name to a document attribute. That way, you could allow it to be overridden. For example:

[source,ruby]
--

Asciidoctor::Extensions.register do |document|

  unless document.attr? 'bibfile'

    document.set_attribute 'bibfile', 'biblio.bib' # <1>

  end
  preprocessor AsciidocBibExtension

end

--

<1> Set the bibfile attribute on the document.

The document may still override this value, unless the attribute was assigned via the CLI or API

-Dan

On Sun, Oct 27, 2013 at 3:53 PM, Dan Allen <[hidden email]> wrote:

Welcome Peter! We're glad to have you!

I'm thrilled to hear that you're working on an integration between Asciidoctor and asciidoc-bib. I've been looking forward to these two projects coming together. I was hopeful the extensions API would open that door, and it seems it has.

The preprocessor is the trickiest extension point to understand at first because it has the ability to affect the reader's cursor. Nothing some documentation can't solve :) So let's get to it!

You have two options when reading the lines in a preprocessor. You can either work with the raw lines of the main source document, or you can instruct the reader to walk the lines.

The second argument to the process method is the raw lines (original source) of the main document. Working with these lines is the most benign since it doesn't affect the reader's cursor. However, as you observed, preprocessor directives in those lines are not processed.

When you read the lines from the reader, any preprocessor directives, such as the include directive, will get processed. It will also advance the cursor, effectively discarding those lines. As such, you need to restore the lines by pushing them back onto the reader. You can so using the Reader#push_include method.

If the process method returns nil, Asciidoctor will continue processing using the reader in its current state. If you return a reader, it will replace the reader Asciidoctor uses.

Here's an example that shows the difference between the raw lines and lines read from the reader.

[source,ruby]
--

class SamplePreprocessor < Asciidoctor::Extensions::Preprocessor
  def process reader, raw_lines
    puts '.Raw lines'
    puts '....'
    puts raw_lines.join # <1>
    puts '....'
    puts "\n"
    lines = []
    while reader.has_more_lines?
      lines << reader.read_line # <2>
    end
    puts '.Preprocessed lines'
    puts '....'
    puts lines.join # <3>
    puts '....'
    use_push_include = true
    if use_push_include
      reader.push_include lines, '<stdin>', '<stdin>' # <4>
      nil # <5>
    else
      Asciidoctor::Reader.new lines # <6>
    end
  end
end
--

<1> Prints the original source of the main document

<2> Consume each line from the reader. The line read is the next line once preprocessor directives on the original line have been evaluated.

<3> Prints the effective lines once all preprocessor directives have been evaluated.

<4> Push the lines back onto the reader so that they can be interpreted by the AsciiDoc processor.
The second and third arguments (file and path, respectively) are required at the moment to work around a bug.

<5> A nil return instructs Asciidoctor to continue with the original Reader

<6> Replace the original Reader with a new one that contains the preprocessed lines

The only major caveat right now with reading the lines from the Reader is that it messes up the line information for reporting warnings. When you push the preprocessed lines back onto the Reader, Asciidoctor thinks they were all in the original file. I'm working on a way to retain this information in the next version.

If you are only reading information from the processed lines, and not modifying them, it's possible to push the raw lines back onto the reader, thus restoring its original state.

[source,ruby]
--

def process reader, raw_lines

  lines = reader.read # <1>

  cursor = reader.cursor
  reader.push_include raw_lines, (cursor.file || '<stdin>'), (cursor.path || '<stdin>'), 1 # <2>
  reader.process_lines = true if cursor.file.nil? # <3>
  nil

end
--

<1> Preprocess all lines and return as an Array

<2> Restore the original lines onto the Reader

<3> Work around an assumption that lines should not be processed if the file name is nil

I hope that clears some things up. Obviously, the Preprocessor extension needs some polishing, but that's what this discussion is helping to identify.

Cheers,

-Dan

On Sun, Oct 27, 2013 at 6:06 AM, plane [via Asciidoctor :: Discussion] <[hidden email]> wrote:

Hi list,

I'm trying to use the extension preprocessor to transform the lines in an asciidoc file prior to processing.  Although I have something working, I find 'include::' macros are not getting processed, and am not sure how to proceed - I'm perhaps not understanding the rendering process correctly.

I'm currently using the 'process' method in a subclass of preprocessor, and changing or inserting lines in the 'lines' array, as appropriate.  I then return a new reader with these new lines.

    def process reader, lines
      return reader if lines.empty?

          # change and insert new rows in 'lines'

      return Asciidoctor::Reader.new lines
    end

This works well with a self-contained asciidoc file, but if the file contains 'include::' macros, these macros are not processed, and their text is simply inserted in the output html without rendering the document to insert.  I have tried 'image::' macros, and these are processed and the image added correctly.

Is the preprocessor an appropriate way to do a source-to-source transform?  I was hoping to slot my rewrite of the asciidoc document in as a first stage, aiming to return a set of lines in correct asciidoc syntax for later rendering.

The code I'm working on is here.

  thanks for any help,

       Peter.

---

If you reply to this email, your message will be added to the discussion below:

http://discuss.asciidoctor.org/extension-preprocessor-tp886.html

To start a new topic under Asciidoctor :: Discussion, email [hidden email]
To unsubscribe from Asciidoctor :: Discussion, click here.
NAML

--

Dan Allen | http://google.com/profiles/dan.j.allen

--

Dan Allen | http://google.com/profiles/dan.j.allen

--

Dan Allen | http://google.com/profiles/dan.j.allen