Inserting toc entries from hyperlinked source code

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Inserting toc entries from hyperlinked source code

daveb
Great progress on Asciidoctor - well done.

I have some html that is auto generated and it has headings I want to include in the toc.  I see I can insert suitable html in to the toc section as a post process extension but is there a better/alternative way of achieving this?

The auto generated html is a source listing and I want to include the function names in the toc.  In addition to updating the toc I will need to add anchors to the function names and I would also like to have the function calls hyperlinked to the function definitions as well.  I am still considering the best way to do this and I think my options are:
       
1. Use a syntax highlighter (coderay or rouge) to generate the html then using the known list of functions edit the html to add in the anchor markup.  This could be done as an asciidoctor extension or standalone process and just use a passthrough include.

2. Extend coderay or rouge to do this linking as part of its syntax highlighting.

3. Post process the html asciidoctor generates either as a post process extension or as an additional step in my doc build process.

Any input welcome.

Dave.
Reply | Threaded
Open this post in threaded view
|

Re: Inserting toc entries from hyperlinked source code

mojavelinux
Administrator

Welcome Dave! And thanks for your encouragement.

Your use case is precisely the kind of customization we set out to make possible with Asciidoctor. There are numerous ways to tap into the processor to get the output that you need, several of which you mentioned.

What's important about the Asciidoctor architecture is that you are never far from being able to plug in code using a regular programming language (primarily Ruby at this point).

One way to enhance the toc output is a post processor. Another way is to provide a custom "view" template (either document, block_preamble or block_toc depending on where you are inserting the toc). You can enhance the listing output in a similar way by providing a custom block_listing template.

You can see examples of custom templates in the https://github.com/asciidoctor/asciidoctor-backends repo.

Keep in mind that you can use Ruby code inside a template, so there is no limit to the complexity of the logic. That's such a core principle of Asciidoctor.

To extract information about what to add to the toc, you might think about using a tree processor in which you can walk the nodes in the parsed tree to extract information. There isn't yet a storage bin in the document object to stash extracted information (expect maybe the @references field), but you could use a (dare I say) global variable in the interim.

I think you it's a good idea to leverage CodeRay or Rouge in both the tree processor and the custom template to handle parsing of the code.

I recommend trying to work within Asciidoctor rather than using an extra step in the doc processing pipeline. The reason is because Asciidoctor provides proper context & comprehension of the document structure and because it was designed for this sort of thing.

If you find that Asciidoctor is not able to give you the control you need at some point, file an issue and we'll work together to improve Asciidoctor so it does.

I hope that helps! If you need help making your customizations, stick them in a GitHub repository and we can review them, time permitting.

Happy hacking!

-Dan

Reply | Threaded
Open this post in threaded view
|

Re: Inserting toc entries from hyperlinked source code

daveb
In reply to this post by daveb
Thanks for the input Dan.  I will try going down the extension route.

My first stab is to replicate the command line functionality I am currently using, but I have run into 3 problems.
1.  The default style sheet isn't being copied into my output.html file.
2.  The include::{build_dir}/interfaces.adoc[] line is being ignored and only a hyperlink to the include file is being inserted into the output html file.
3.  The syntax highlighting is being ignored and nothing is being inserted into the output html file.
               [source, c]
               ----
               include::./urb_update.cpp[]
               ----

My command line (in rake) is:

sh " asciidoctor -d book -n -a data-uri -a toc2 -a toclevels=4 -a source-highlighter=coderay -a build_dir=#{UNIT_BUILD_DIR} #{input_file}"

and my asciidoctor runner is:

require 'asciidoctor'

def asciidoctor_runner (file, attributes = {})
       
        attributes.merge!( 'doctype'            => 'book',
                                                'source-highlighter' => 'coderay',
                                                'data-uri'            => true,
                                                'toc2'                  => true,  
                                                'toclevels'          => 4,
                                                'numbered'       => '',
                                                'linkcss'  => true
                                        )

       
        Asciidoctor.render_file(file, :in_place => true, :backend => 'html', :attributes => attributes)
end


and I call this from my rake task:

asciidoctor_runner(input_file, 'build_dir' => UNIT_BUILD_DIR)


No errors are printed out by asciidoctor.

I think the section in the docs that talks about the API could be expanded to say if this is your command line invocation then this is the equivalent API invocation.

Dave,
Reply | Threaded
Open this post in threaded view
|

Re: Inserting toc entries from hyperlinked source code

mojavelinux
Administrator
Dave,

Aha! There's a very important piece of information you need to know, which we'll be sure to call attention to in the user guide. When you switch from the CLI (asciidoctor) to the API, the default safe mode switches from the lowest (unsafe) to the highest (secure). We kept the CLI consistent with AsciiDoc Python, but the API default had to change to accommodate requirements for GitHub deployment.

To restore all the missing functionality, add the following option to the Asciidoctor#render_file invocation:

:safe => :unsafe

or

:safe => :safe

That should solve #1, #2 and #3.

To help highlight this change, we'll definitely add the API equivalent of what a given command line invocation does. Good thinking! 

The main (and really only) difference between the unsafe and safe modes is that in unsafe mode, Asciidoctor can include files from anywhere, whereas in safe mode it can only include files starting from the working directory.

Btw, in AsciiDoc, the attribute value empty string is the same as a true value. Internally, Asciidoctor only looks to see if the key is set in the case of a boolean attributes.

-Dan


On Thu, Oct 3, 2013 at 4:42 AM, daveb [via Asciidoctor :: Discussion] <[hidden email]> wrote:
Thanks for the input Dan.  I will try going down the extension route.

My first stab is to replicate the command line functionality I am currently using, but I have run into 3 problems.
1.  The default style sheet isn't being copied into my output.html file.
2.  The include::{build_dir}/interfaces.adoc[] line is being ignored and only a hyperlink to the include file is being inserted into the output html file.
3.  The syntax highlighting is being ignored and nothing is being inserted into the output html file.
               [source, c]
               ----
               include::./urb_update.cpp[]
               ----

My command line (in rake) is:

sh " asciidoctor -d book -n -a data-uri -a toc2 -a toclevels=4 -a source-highlighter=coderay -a build_dir=#{UNIT_BUILD_DIR} #{input_file}"

and my asciidoctor runner is:

require 'asciidoctor'

def asciidoctor_runner (file, attributes = {})
       
        attributes.merge!( 'doctype'            => 'book',
                                                'source-highlighter' => 'coderay',
                                                'data-uri'            => true,
                                                'toc2'                  => true,  
                                                'toclevels'          => 4,
                                                'numbered'       => '',
                                                'linkcss'  => true
                                        )

       
        Asciidoctor.render_file(file, :in_place => true, :backend => 'html', :attributes => attributes)
end


and I call this from my rake task:

asciidoctor_runner(input_file, 'build_dir' => UNIT_BUILD_DIR)


No errors are printed out by asciidoctor.

I think the section in the docs that talks about the API could be expanded to say if this is your command line invocation then this is the equivalent API invocation.

Dave,


If you reply to this email, your message will be added to the discussion below:
http://discuss.asciidoctor.org/Inserting-toc-entries-from-hyperlinked-source-code-tp668p681.html
To start a new topic under Asciidoctor :: Discussion, email [hidden email]
To unsubscribe from Asciidoctor :: Discussion, click here.
NAML



--
Reply | Threaded
Open this post in threaded view
|

Re: Inserting toc entries from hyperlinked source code

daveb
In reply to this post by daveb
Dan,  that did the trick.  I now have my extensions working and if anyone is interested they are below.  I have updated to toc with a postprocessor extension which is a little fragile and could break if the toc markup changed.  It would seem reasonable for a block macro to be able to add entries into the toc directly (maybe it can and I just didn't find out how).

Dave.


require 'asciidoctor'
require 'asciidoctor/extensions'
require 'coderay'

# Easy, if clunky, way of passing data between extensions.
$headings = {}

Asciidoctor::Extensions.register do |document|
    block_macro :bcode, BSourceCode
    postprocessor CustomTocPostprocessor
end

# Find all the function definitions in the target file and after it has been syntax
# highlighted make the functions hyperlink targets.  Make all calls to these functions
# hyperlinks and store the function names in a global for later use.
class BSourceCode < Asciidoctor::Extensions::BlockMacroProcessor
    def process (parent, target, attributes)
        src_dir = @document.attributes.fetch('src_dir', '')
        src_file = File.join(src_dir, target)
        css = @document.attributes.fetch('coderay-css','class').to_sym
        source = CodeRay::Duo.new(:c, :html, :css => css).highlight(File.read(src_file))
   
        functions = []
        prefix = File.basename(target)
           
        # Find all the function definitions and add anchors to them.
        source.gsub!(/(.*?<\/span>\s*)(\w+)/) do
            functions << $2
            $1 + "#{$2}"      
        end
   
        # Find all calls to the functions and make them into hyperlinks.
        source.gsub!(/(?<![_>])(#{functions.join('|')})/, '<a href="#' + prefix + '_\1">\1')
       
        content = %Q{
            <div class="listingblock">
                <div class="content">
                    <pre class="CodeRay"><code class="c language-c"> 
                    #{source}
                </div>
            </div>
        }
       
        # Record the headings we want to add into the toc.
        $headings[parent.name] = {:prefix => prefix, :functions => functions}

        Asciidoctor::Block.new parent, :pass, :content_model => :raw, :source => content    
    end
end

# Add all the functions we have hyperlinked into the toc.
class CustomTocPostprocessor < Asciidoctor::Extensions::Postprocessor
    def process(output)
        $headings.each do |name, headings|
            prefix = headings[:prefix]
            section = %Q{<ul class="sectlevel2">\n} +
                        headings[:functions].map {|f| %Q{<li><a href="##{prefix}_#{f}">#{f}<\/a><\/li>\n}}.join
            output = output.sub(/^(<li><a href=".*?#{name}<\/a><\/li>)$/, '\1' + section)
        end
       
        output  
    end
end

def asciidoctor_runner (file, attributes = {})
   
    attributes.merge!(  'doctype'            => 'book',
                        'source-highlighter' => 'coderay',
                        'data-uri'           => true,
                        'toc2'               => true,  
                        'toclevels'          => 4,
                        'numbered'           => true,
                        'linkcss'            => true
                    )

   
    Asciidoctor.render_file(file,   :in_place => true,
                                    :backend => 'html',
                                    :safe => :unsafe,       # just allow access to any dir for includes
                                    :attributes => attributes)
end