Asciidoctorj API questions

Hello!

I've decided to invest some time into writing Java extension that will do the following (well, it actually does that already, but the integration part is a bit ugly).

Given this source

[tree]
> A
>> B
>>> C
>>> D
>> E
>>> F
>> G
>> H

it produces this

A
├── B
│   ├── C
│   └── D
├── E
│   └── F
├── G
└── H

Project in it's current state is available here: https://github.com/allati/asciidoctor-extension-tree

I'm not confident this extension will be useful in practice, though. However, it was pretty interesting to write one :) But the purpose of this post is not about what this extension does, but more about what problems I've faced when trying to integrate it with Asciidoctor. I hope that you will take time to answer my questions and clarify some points in the API. I apologize for not writing all the questions at once - I'm not sure that I will bring myself to thoroughly write about all problems in one post and even if I would - that anyone would read it to the end anyway :)

Now, back to the point.

JavaExtensionRegistry provides several ways to register Block extension - one of them registers BlockProcessor instance, and the other one registers it by class. I had success with the former, but with the latter I always get the following error (line numbers might be a little off, because I was adding "puts" here and there):

Exception in thread "main" org.jruby.exceptions.RaiseException: (ArgumentError) Invalid arguments specified for registering block extension: [nil, :tree, {}]
	at RUBY.add_syntax_processor(.../asciidoctorj-1.5.2.jar!/gems/asciidoctor-1.5.2/lib/asciidoctor/extensions.rb:1212)
	at RUBY.block(.../asciidoctorj-1.5.2.jar!/gems/asciidoctor-1.5.2/lib/asciidoctor/extensions.rb:889)
	at RUBY.block_processor(<script>:51)
	at org.jruby.RubyBasicObject.instance_exec(org/jruby/RubyBasicObject.java:1562)

Can you please suggest what I'm doing wrong?

~ Regards, Vadim.

Welcome Vadim,

It’s always nice to have new people playing with the API :)
First of all, you will see from the maven and gradle plugin’s documentation that such an extension would in fact be very useful:
https://github.com/asciidoctor/asciidoctor-maven-plugin
https://github.com/asciidoctor/asciidoctor-gradle-plugin
A shame that GitHub does not allow to integrate extensions (at least for now).

Now, back to the issue. Based on your code I think the problem may be related to the initialization of the `config` map. You are initializing it outside the processor https://github.com/allati/asciidoctor-extension-tree/blob/master/src/main/java/com/github/allati/asciidoctor/tree/AsciidoctorTree.java#L24-25, so that when passing a class those values are not set. Imho, if possible, it’s best to initialize it inside the processor itself to avoid errors when injected as a class and not and instance.

For more details you’ll find a simple example I wrote for testing:
https://github.com/abelsromero/asciidoctor-maven-plugin/blob/be195ab763b7a219633febfeb46c40b03108c92c/src/test/java/org/asciidoctor/maven/test/processors/YellBlockProcessor.java

Please, let me know if this answers your question.

I was actually looking at passing the output of tree to imagemagik the
other day, for representations of file structures, so I'd say there is some
interest. ;-)

Thanks for such a warm welcome! :)

abelsromero wrote

Based on your code I think the problem may be related to the initialization of the `config` map. You are initializing it outside the processor https://github.com/allati/asciidoctor-extension-tree/blob/master/src/main/java/com/github/allati/asciidoctor/tree/AsciidoctorTree.java#L24-25, so that when passing a class those values are not set.

I'm afraid it is not the case, because class constructors are not being invoked at all - it seems that the exception is thrown before Asciidoctor has a chance to try to create Processor instance. I should have made this point clear when describing the problem.

abelsromero wrote

Imho, if possible, it’s best to initialize it inside the processor itself to avoid errors when injected as a class and not and instance.

As far as I understand, org.asciidoctor.extension.Processor converts config map to Ruby representation only when this map is passed to it's constructor. Since superconstructor must be the first statement in a subclass constructor, it is necessary to provide some static method that will prepare configuration map as a part of superconstructor invocation, like

private static class MonoTreeProcessor extends BlockProcessor {
	public MonoTreeProcessor() {
		super("tree", prepareConfig());
	}

	private static Map<String, Object> prepareConfig() {
		Map<String, Object> config = new HashMap<>();
		config.put("contexts", Arrays.asList(Ruby.getGlobalRuntime().newSymbol("listing")));
		return config;
	}
	...
}

Combined with annotation processing Dan mentioned (if this configuration map will remain), it may become a problem, because initial config must be created before injection can be performed (if any). Maybe it worth creating a notion of extension lifecycle? In this case Java -> Ruby parameter conversion can be done in the "post-init" phase, for example, and any configuration activities (like populating config map) can be safely done after extensions are instantiated/fields injected, but before first rendering is performed.

~ Regards, Vadim

I've been experimenting and indeed, as you say, the constructors are not invoked when passing a class in your code. However I found a way to make it work.

Just a note before the fix, I found out that using a private static class does not seem to work and the following error is thrown.

org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `MonoTreeProcessor' for Java::ComGithubAllatiAsciidoctorTree::AsciidoctorTree:Class
        at RUBY.(root)(<script>:1)

It seems to me that the runtime is not able to find a constructor and fails. To confirm if this a asciidoctorj problem I'd need to further look into it. You can give it a try if you like and see if you can confirm it.

Now, and here comes the solution. Several things in the code seems to cause issues to asciidocotor, this does not mean you're code is wrong, more on the contrary, I think this means that the extensions SPI and documentation needs some more work.
Having said so, here is the list of things:
- More than one constructor does cause problems -> Just remove all constructor but the config's one.
- Initialization using RubyObjects does fail too -> Adding a initialization using just ":listing" (String) works.

Here is my final code, note I changed the name of the class in my example and this works using all three configurations options: instance, class and class name.

    @SuppressWarnings("serial")
    private static final Map<String, Object> configs = new HashMap<String, Object>() {{
        put("contexts", Arrays.asList(":listing"));
    }};

    public MyBlockProcessor(String name, Map<String, Object> config) {
        super(name, configs);
    }

I hope this helps.

BTW, this extension lifecycle is something we should take into consideration.

It definitely helped a lot! Thanks! Public class with only one constructor was successfully picked up and executed. It was equally successful if initialized with RubyObjects, but ":name" syntax is much more concise, so I'll stick to it. It's time to move on to the other questions, I guess :)

1. What is the preferred way of getting a list of available configuration options for a given Processor type? By trial and error I've figured out that "contexts" option is required for BlockProcessor, but I'm sure there are many other options.

2. Where can I find a list of valid context names?

3. Is there a way to find out in BlockProcessor#process(AbstractBlock, Reader, Map<String, Object>) context (type) of the block that is being processed? For example, if I add this configuration

configuration.put("contexts", Arrays.asList(":listing", ":paragraph"))

extension will be associated with two different contexts. Will I be able to produce one output for paragraph and another output for listing?

4. Can I register BlockProcessor to alter or override standard asciidoc block rendering? I originally had thought to declare this "tree" as the additional style for a list. For example,

[disk]
. A
.. B
. C

will be rendered as a bullet list, but

[tree]
. A
.. B
. C

will be rendered as a tree. I guess it can be done with TreeProcessor, but BlockProcessor looks like more "focused" way of handling specific blocks.

~ Regards, Vadim

First of all, keep in mind that Java extensions are functional but are still a “work in progress” thing and documentation is currently short. In fact, most of the changes in the current DEV branch of AsciidoctorJ have been done to improve extension (btw, kudos to Robert Panzer).
Having said that, questions like yours are really appreciated and useful to us :)

Back to your questions, I can provide some insight about 1 and 2. Not so for 3 and 4, but I hope you can experiment a bit with the information on the information below.

vdmitriev wrote

1. What is the preferred way of getting a list of available configuration options for a given Processor type? By trial and error I've figured out that "contexts" option is required for BlockProcessor, but I'm sure there are many other options.

2. Where can I find a list of valid context names?

You can find some information in these links:
https://github.com/asciidoctor/asciidoctor-documentation/issues/30 <-- Specially this one
http://discuss.asciidoctor.org/process-method-in-Java-block-extension-not-being-executed-td2742.html
http://asciidoctor.org/docs/user-manual/#extensions <-- even if you don't know Ruby, the code it's quite expresive

These links will certainly prove useful. Thanks for all the tips!

~ Regards, Vadim

Hi,

Thanks for your monotree extension, it's nice and simple. I am using it in my documentation, but I needed to display links in this tree. As the block created is a "listing" block, my links weren't interpreted by Asciidoctor.

That's why I modified a little your extension to render a block of the type "example"  which renders a normal Asciidoctor substitution, that interprets links, special characters... I dealt with the blank spaces/line breaks issues.

My manager now wants me to ask a pull request to you, so that we can fetch the extension from its github. But I would have to put 2 contexts in the config: listing and example, in order to keep both options for the tree, something like:
[tree, configcontext= "listing"]
[tree, configcontext= "example"]
Then get the code to process differently according to the block context.

This actually corresponds to the 3d question you asked earlier:

 

vdmitriev wrote

3. Is there a way to find out in BlockProcessor#process(AbstractBlock, Reader, Map<String, Object>) context (type) of the block that is being processed? For example, if I add this configuration

configuration.put("contexts", Arrays.asList(":listing", ":paragraph"))

extension will be associated with two different contexts. Will I be able to produce one output for paragraph and another output for listing?

Do you have an answer for this question ?

Thanks a lot,

Clara

PS: the very little change I made to get an Asciidoctor-interpreted yellow block are:
- In symbolsets.properties

simple.EMPTY = {nbsp}{nbsp}{nbsp}{nbsp}
simple.PASSTHROUGH = {vbar}{nbsp}{nbsp}{nbsp}{nbsp}
simple.JUNCTION = {plus}--{nbsp}
simple.TERMINAL = {backtick}--{nbsp}

fancy.EMPTY = {nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}
fancy.PASSTHROUGH = \u2502{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}
fancy.JUNCTION = \u251C\u2500\u2500\u2500{nbsp}
fancy.TERMINAL = \u2514\u2500\u2500\u2500{nbsp}

- In MonotreeProcessor.java, replace "listing" by "example" in the config and in the createBlock() parameters

- In MonotreeProcessor.java, process method :

 attributes.put("hardbreaks", true); 

 
PPS: this last thing (set the hardbreak attribute to true) won't work, I'm forced to declare trees with

 [tree, options="hardbreaks"]

 to get line breaks...

Hi, Clara!

First a little sidenote: I called this extension monotree, because it relies heavily on the fact that monospace fonts have all the characters of the same width. It ensures that no matter which monospace font you have configured (courier, droid mono, consolas, etc.) - the tree will always be lined up properly.

'Example' sections have normal font and these fonts generally have spaces much narrower than other characters and I guess that it was the reason why you increased number of spaces for empty and passthrough elements - otherwise monotree output looks misaligned.

fancy.EMPTY = {nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}
fancy.PASSTHROUGH = \u2502{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}{nbsp}

Here you have 11 spaces instead of original 4 to compensate for width loss of the 'space' character. Though your solution definitely works, it heavily depends of the font you are using now. Should you change it - tree alignment will likely break once again.

If your only requirement is to render links within a tree - Dan made an excellent suggestion. Simply add subs=normal and links will render:

[tree, subs=normal]
----
> http://asciidoctor.org
----

Now for the second part of your question regarding different rendering for different section types. There is no API to infer section type within BlockProcessor#process. My solution was to create separate block processor for every section type (possibly with some common supertype). But after finding this workaround I faced another rendering-related issue. To have consistent output for any backed, I need a way to apply monospace font. But if I do it asciidoc way (using grave accent markup chars), the result is not ideal:



I need a way to get rid of grey boxes, but it is unlikely that there is any.

~ Vadim

P.S. I'm sorry for answering you a month later, but it was a wild month. Your interest in my extension is flattering and I really hope that I can help you with your task.