Administrator
|
This post was updated on .
Ever since I started working on Asciidoctor, I've been eagerly awaiting the chance to work on the extensions API. That time has come. Extensions are the last major blocker for the 0.1.4 release.
I've been looking forward to this enhancement because extensions in AsciiDoc (technically called filters) are very powerful, but also very low-level. What's worse, they rely on system commands to do anything significant. The goal for Asciidoctor has always been to allow extensions to be written using the full power of a programming language (whether it be Ruby, Java, Groovy or JavaScript), similar to what we've done with backend (rendering) mechanism. That way, you don't have to do yak shaving to get the functionality you want and you can distribute the extension using defacto-standard packaging mechanisms (like RubyGems or JARs). I've pushed a draft of the API as a pull request for issue #79 (https://github.com/asciidoctor/asciidoctor/issues/79), one of several issues related to extensions. WARNING: The extension API should be considered experimental and subject to change, even after it's integrated into the 0.1.4 release. It's likely going to take several iterations to get right and we don't want to lock ourselves into an API too early. Early adopters will help us identify and close the gaps and smooth out any awkwardness. TIP: For those of you on the JVM, yes, you can write extensions in Java. I've prototyped it and it works. We'll need to sort out a few technical challenges and documentation to make it completely smooth, but we'll get there. Part of that is creating abstract extension classes in the Java integration that mirror the classes in the Ruby API...though if you're starting from Groovy that may not even been necessary. Conceptually, the extensions themselves are rather simple. The hard part came in figuring out how they would be registered in a non-static way and how to weave them into the processor. After studying many other libraries (most notably Middleman, Python Markdown and Kramdown), I think I finally sorted out an elegant approach. Here's a list of the extension points we have so far. Preprocessor:: Processes the raw source lines before they are passed to the parser Treeprocessor:: Processes the Document (AST) once parsing is complete Postprocessor:: Processes the output after the Document has been rendered, before it's gets written to file Block processor:: Processes a block of content marked with a custom style (i.e., name) (equivalent to filters in AsciiDoc) Block macro processor:: Registers a custom block macro and process it (e.g., gist::12345[]) Inline macro processor:: Registers a custom inline macro and process it (e.g., btn:[Save]) Include processor:: Processes the include::<filename>[] macro NOTE: The include processor extension is in a separate pull request and will use the same extension mechanism. These extensions are registered per Document using a callback that feels sort of like a DSL: ```ruby Asciidoctor::Extensions.register do |document| preprocessor SamplePreprocessor treeprocessor SampleTreeprocessor postprocessor SamplePostprocessor block :sample, SampleBlock block_macro :sample, SampleBlockMacro inline_macro :sampe, SampleInlineMacro end ``` Each class registered is instantiated when the Document is created. You can register more than one processor for each type, though you can only have one processor per custom block or macro. I also plan to have an extension point that can process one of the built-in blocks, such as a normal paragraph. I just haven't sorted out the best way to integrate it yet. For now, you need to use the Asciidoctor API (not the CLI) in order to register the extensions and invoke Asciidoctor. Eventually, we'll be able to load extensions packaged in a RubyGem (Ruby) or JAR (Java) by scanning the LOAD_PATH (Ruby) or classpath (Java), respectively. We may also ship some built-in extensions that can be enabled using an attribute named `extensions`. Here are some examples of extensions and how they are registered. == Preprocessor example Remove front-matter required by site generators like Jekyll. ```ruby require 'asciidoctor/extensions' # Sample document: # # --- # tags: [announcement, website] # --- # = Document Title # # content # class FrontMatterPreprocessor < Asciidoctor::Extensions::Preprocessor def process lines return lines if lines.empty? front_matter = [] if lines.first.chomp == '---' original_lines = lines.dup lines.shift while !lines.empty? && lines.first.chomp != '---' front_matter << lines.shift end if (first = lines.first).nil? || first.chomp != '---' lines = original_lines else lines.shift @document.attributes['front-matter'] = front_matter.join.chomp end end lines end end Asciidoctor::Extensions.register do |document| preprocessor FrontMatterPreprocessor end Asciidoctor.render_file('sample-with-front-matter.ad', :safe => :safe, :in_place => true) ``` TIP: The processor can be a Java class (see below) == Block processor example Register a custom block style that uppercases all the words and converts periods to exclamation points. ```ruby # Sample: # # [yell] # The time is now. Get a move on. # class YellBlock < Asciidoctor::Extensions::BlockProcessor option :contexts, [:paragraph] option :content_model, :simple def process parent, reader, attributes lines = reader.lines.map {|line| line = line.upcase line = line.gsub('.', '!') line } Asciidoctor::Block.new(parent, :paragraph, :source => lines, :attributes => attributes) end end Asciidoctor::Extensions.register do |document| block :yell, YelloBlock end Asciidoctor.render_file('sample-with-yell-block.ad', :safe => :safe, :in_place => true) ``` == Block macro processor example Add a macro for embedding a gist. ```ruby # Sample: # # gist::123456[] # class GistMacro < Asciidoctor::Extensions::BlockMacroProcessor def process parent, target, attributes if @document.attr? 'basebackend', 'html' Block.new(parent, :pass, :content_model => :raw, :source => %(<div class="gistblock"> <div class="title">#{attributes['title']}</div> <div class="content"> <script src="https://gist.github.com/#{target}.js"></script></div> </div>)) else # TODO Download gist and convert it to a listing # For now, just create as a link (subject to normal paragraph substitutions) Asciidoctor::Block.new(parent, :paragraph, :source => %(https://gist.github.com/#{target}"), :attributes => attributes) end end end Asciidoctor::Extensions.register do |document| block :gist, GistBlock end Asciidoctor.render_file('sample-with-gist.ad', :safe => :safe, :in_place => true) ``` == Java-based preprocessor example .Extension class from Asciidoctor Java (planned) ```java import java.util.List; public abstract class Preprocessor { private Object document; public Preprocessor(Object document) { this.document = document; } public abstract List<String> process(List<String> lines); } ``` ```java import java.util.List; public class JavaBasedPreprocessor extends Preprocessor { public JavaBasedPreprocessor(Object document) { super(document); } public List<String> process(List<String> lines) { System.out.println("process method called on " + getClass().getName() + " instance with lines:"); System.out.println(lines); // remove first line for fun lines.remove(0); return lines; } } ``` javac Preprocessor.java JavaBasedPreprocessor.java ```ruby $using_jruby = RUBY_ENGINE == 'jruby' if $using_jruby require 'java' java_import 'JavaBasedPreprocessor' end Asciidoctor::Extensions.register do |document| preprocessor JavaBasedPreprocessor if $using_jruby end Asciidoctor.render_file('sample.ad', :safe => :safe, :in_place => true ``` We need your feedback! Please post your reply here or comment on issue #79. -- Dan Allen | http://google.com/profiles/dan.j.allen |
Not sure why you're stubbing out the method in the processors. Sure the initialize is fine, but process? Not really needed, it would always do nothing at worst, but you could always check for a responds_to(:process). You can add in params there too. If it doesn't respond to that, well, it isn't an extension. Also using process seems like it'll make it cumbersome to uses lambdas or Procs as extensions, because they only have a call method. I suppose you could wrap a Proc/lambda in an object, but that seems more pain than it's worth.
On Thu, Aug 15, 2013 at 1:52 PM, mojavelinux [via Asciidoctor :: Discussion] <[hidden email]> wrote:
|
Administrator
|
This post was updated on .
In reply to this post by mojavelinux
When you are creating a custom macro, it's pretty typical to also require a custom rendering template. You can, of course, create one using a template language supported Tilt, put it into a directory, then pass that directory to the `template_dir` option of Asciidoctor. However, that would mean that anyone using your extension would need to remember to do this (for now).
A better approach is to include the template with the extension code. It's possible to define a template in a block and register that with the document when the extension is initialized. Here's an example that uses a custom "p" template. ```ruby require 'tilt' block_p_template = Tilt.new('block_p.html.slim') do |t| "p=content" end # Sample content # # [simple] # Simple paragraph # class SimpleParaBlock < Asciidoctor::Extensions::BlockProcessor option :contexts, [:paragraph] option :content_model, :simple def initialize(document) super document.renderer.register_view 'block_p', block_p_template end def process parent, reader, attributes Asciidoctor::Block.new(parent, :p, :source => reader.lines) end end Extensions.register do |document| block :simple, SimpleParaBlock end Asciidoctor.render_file('sample-with-simple-para.ad', :safe => :safe, :in_place => true) ``` -- Dan Allen | http://google.com/profiles/dan.j.allen |
Administrator
|
...which would output:
```html <p>Simple paragraph</p> ``` -Dan |
Administrator
|
In reply to this post by mojavelinux
I just sent a new pull request for issue #100, which is the custom include macro processor. Here's an example of how it can be used.
```ruby require 'uri-open' # Sample content: # # include::https://raw.github.com/asciidoctor/asciidoctor/master/Gemfile[] # class RemoteIncludeProcessor < Asciidoctor::Extensions::IncludeProcessor def handles? target target.start_with?('http://') or target.start_with?('https://') end def process target, attributes open(target).readlines end end Asciidoctor::Extensions.register do |document| include_processor RemoteIncludeProcessor end Asciidoctor.render_file('sample-with-include.ad', :safe => :safe, :in_place => true) ``` -Dan |
In reply to this post by mojavelinux
Hi Dan,
I have been thinking about how to implement extensions in Asciidoctor Java Integration. The first part, the easy one is about registering the extensions, for this case, I will implement a Registry class (more or less like Asciidoctor) which will deal with registration part of Java classes, it will be composed by one Ruby class and one Java interface. But where I find a lot of work is in some kind of extensions: for example a preprocessor is very easy, it receives a list of Strings and returns a list of Strings, no problem because conversion is done by JRuby. But for example in case of BlockMacroProcesor, there are more attributes which we should inspect and possibly bind manually to Java interfaces. There is no problem in doing in that way but it will require some effort of manual mapping. Moreover to avoid Asciidoc problem with Filters (aka Extensions) of calling directly CLI, we will need to provide a Block class in Java so users can use it too to create new blocks. Another problem will be how to register Ruby extensions directly from Asciidoctor Java Integration, I think that with a simple overload of register method would be enough. But for now I will release a preview version where from Java part you will be only available to register Java extensions. Now that I have things more clear, it is time to code a bit :D. Only last thing when have you planned to merge pull request to master branch? |
Dan I have just coded a first draft of extensions for AJI but I have some questions or a behavior which I am not pretty sure it is normal, let me show in code, maybe you find exactly the problem:
I have created a method which acts as a proxy to a Ruby class (the Asciidoctor Java class): ```ruby require 'java' require 'asciidoctor/extensions' // some code for rendering documents def preprocessor(processor) puts processor Asciidoctor::Extensions.register do |document| puts "Hi" preprocessor processor end end ``` 'processor' is a java.lang.String which in fact would be the full qualified name of the extension, for example: org.asciidoctor.extensions.SamplePreprocessor The problem is that when I execute the preprocessor method, the first puts prints the full qualified name of the class, but the puts "Hi" is never shown. I could understand that the String that is send to Asciidoctor is not treated correctly and we should think other possibilities (this is not a problem), but maybe I am doing something wrong? Of course this is a test that will be refactored to an ExtensionRegistry class. |
Administrator
|
In reply to this post by mojavelinux
I'm happy to announce that the pull requests for the extensions have been merged into master! And that means the ball is rolling on the 0.1.4 release. I'm planning on pushing a release candidate on Fri or Sat and hopefully a release at the beginning of the week. As it turned out, I had to do a pretty massive overhaul of how Asciidoctor was reading files in order to make the behavior more consistent once extensions were added to the mix. I'll include details in the release notes about what changed, why and what it means. With the rework of the reader came some changes to the extension API. For instance, the IncludeProcessor used to accept an array of lines and return either the same array or a replacement. While simple and straightforward, it's too primitive to work correctly in AsciiDoc. It lacks any context about what changed in the lines and where the new lines originated for things like error reporting, etc. Thus, now the IncludeProcessor gets a Reader object and when it has new content to add, it needs to call Reader#push_include to push the new data onto the stack with the proper context. Here's a sample boilerplate text IncludeProcessor as an example: class BoilerplateTextIncludeProcessor < Asciidoctor::Extensions::IncludeProcessor def handles? target target == 'lorem-ipsum.txt' end def process reader, target, attributes lines = ["Lorem ipsum dolor sit amet...\n"] reader.push_include lines, '<extension>', target, 1, attributes end end I plan to include plenty of other examples in the release notes, blog entry and/or docs. I'll reiterate that these APIs are highly experimental. You should consider working with them if you want to help shape and refine them. My hope is that we can say that they are reasonably stable for the 1.5.0 release (when we finally get out from under to 1.0.0 glass ceiling). |
Administrator
|
In reply to this post by asotobu
Alex, First, that's exciting news! I think the problem you are running into is that I haven't yet put in the code that allows the processor to be specified as a String. I just noted that down on my list and will update the code later today, along with a test. Stay tuned. One of the reasons I'm really excited about the Java integration with extensions is because it will help us work out a contract for these APIs...both the core APIs and the extension APIs. I'm open to all suggestions...we just need to keep bouncing the ideas around. Cheers, On Tue, Aug 20, 2013 at 1:52 PM, asotobu [via Asciidoctor :: Discussion] <[hidden email]> wrote: Dan I have just coded a first draft of extensions for AJI but I have some questions or a behavior which I am not pretty sure it is normal, let me show in code, maybe you find exactly the problem: -- Dan Allen | http://google.com/profiles/dan.j.allen |
Thank you so much and sorry for being so impatient :) I saw that there isn t any method for string but because I am not an expert of Ruby I thought that an automatic change occurs. Well I will update to last code with Strings and a first approach with Preprocessor in Java will be ready
El divendres 23 d’agost de 2013, mojavelinux [via Asciidoctor :: Discussion] ha escrit:
-- Enviat amb Gmail Mobile |
Free forum by Nabble | Edit this page |