Upgrading Asciidoctor.js to the latest version of Asciidoctor

Hi,

I cloned https://github.com/asciidoctor/asciidoctor.js and tried to build it but I had to modify few things :
 * gem install opal-sprockets
 * Add require 'opal-sprockets' in Rakefile
 * Replace syntax "boolean ? yesValue : noValue" in /lib/asciidoctor/lexer.rb around 1570 with :

        
        if value.nil?
          accessible = parent.document.delete_attribute(name)
        else
          accessible = parent.document.set_attribute(name, value)
        end

Then I successfully generated asciidoctor.js with Opal 0.4.4 but I had to manually edit the generated Javascript :

      def.$attr = function(name, default$, inherit) {
        var $a, $b;if (default == null) {

      def.$attr = function(name, default$, inherit) {
        var $a, $b;if (default$ == null) {

"default" parameter was missing a $ character but I have no clue why... I checked the helpers ruby class but there is no parameter named "default".

I was thinking maybe I can open an issue on Opal project but I don't really have much information to provide :(

I created a branch on asciidoctor.js to commit my changes and start working on cross-compiling asciidoctor 0.1.4. If someone is interested you're welcome to help :)

https://github.com/Mogztter/asciidoctor.js/tree/0.1.4
https://github.com/Mogztter/asciidoctor/tree/opal


Guillaume.

Great work Guillaume! You got past the hardest part of any project, which is making the compiler happy :)

It looks like some regressions have snuck into Opal since the last time I compiled asciidoctor.js. Asciidoctor gives Opal a thorough workout, so chances are you've found a real regression. I encourage you to open an issue with Opal and give fixing it a go. The Opal guys (Adam and Elia) are really cool and welcoming. The test suite in Opal is just like a normal Ruby unit test suite, so it's really easy to get running.

I have good news for you as well!!! I just finished updating all the html5 erb templates for Asciidoctor 0.1.4!

You can find the templates in the https://github.com/asciidoctor/asciidoctor-backends/tree/master/erb/html5 repository. These are drop-in replacements for the opalerb files in Asciidoctor.js. Just change the file extension from .html.erb to .opalerb and copy them to lib/backends/html5.

Of course, the templates depend on the 0.1.4 core, so they'll only be useful once you get that part upgraded. Since the changes from 0.1.2 to 0.1.4 are rather significant, I think it's best to look at the diff applied to asciidoctor in asciidoctor.js, understand what needed to be changed, and reapply those changes. That will also help you become familiar with what we need to change upstream to eliminate (or reduce) the number of differences needed.

The changes basically boil down to:

* OPAL constant <= this we can apply upstream for 1.5.0
* addition of a compat library to close gaps in Opal <= this we can apply upstream for 1.5.0
* load libraries explicitly (since everything is statically compiled)
* string modifications must be changed to assignments (cannot use chop! for instance)
* differences in regex character classes <= these we can apply upstream for 1.5.0
* load and use opalerb templates instead of built-in templates
* cannot use Struct (need to use concrete class) <= this we can apply upstream for 1.5.0
* destruct assignments weren't working last I checked in Opal
* regexps don't match endline at end of line, so need to add endlines only during output (this is a significant change, though I'd like to make it in 1.5.0 if possible)

A good first step is to complete this list of incompatibilities so that we have a good understanding of what we are changing. You can keep track of them in the README. Once you that picture in focus, applying the rules goes very quickly...and you can convert just about any Ruby code :)

If you have any questions, please don't hesitate to ask. I'll definitely pitch in to help out whenever I can.

-Dan

--
Dan Allen | http://google.com/profiles/dan.j.allen

Hi both,

Looks like these are both regressions in Opal.

* The missing `$` from `default$` is now fixed on master.

* The broken ternary passing has also been noticed recently, and a fix is underway.

I will post back here once it has been fixed (with proper tests added!).

Thanks,

Adam.

Update:

I have fixed the Opal bug in parsing ternary statements, as well as the example to now use `$global` instead of `$window` as the `Native` instance.

I am now encountering a different bug in running the example, where a variable is `nil` instead of an expected string. I will dig down and see if I can route out the problem.

My fork is here, with the very small change: https://github.com/adambeynon/asciidoctor.js/commit/b05b8fd1b909eed068541fce29c626196452e941.

Adam.

Update 2:

asciidoctor.js is now fully working in my fork, http://github.com/adambeynon/asciidoctor.js, and you must run opal from master (as setup in the Gemfile). There was a bug in `Enumerable#collect` which is now fixed, and a slight change was needed in the example to be compliant with Opals new native sending.

Adam.

Hi Adam,

Thanks for your help :)
I checkout your fork but I still have an error :

$ opal -v
Opal v0.4.4

$ rake dist
Cannot handle dynamic require :asciidoctor:9
Cannot handle dynamic require :asciidoctor:10
Cannot handle dynamic require :asciidoctor/helpers:22
rake aborted!
parse error on value ":" (":") :asciidoctor/lexer:1573
  (in /home/mogztter/workspaces/asciidoctor.js/asciidoctor/lib/asciidoctor/lexer.rb)
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/bundler/gems/opal-5244182bdbb7/lib/opal/lexer.rb:40:in `on_error'
(eval):3:in `_racc_do_parse_c'
(eval):3:in `do_parse'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/bundler/gems/opal-5244182bdbb7/lib/opal/lexer.rb:33:in `parse'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/bundler/gems/opal-5244182bdbb7/lib/opal/parser.rb:75:in `parse'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/bundler/gems/opal-5244182bdbb7/lib/opal/require_parser.rb:17:in `parse'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/opal-sprockets-0.2.0/lib/opal/sprockets/processor.rb:73:in `evaluate'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/tilt-1.4.1/lib/tilt/template.rb:103:in `render'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/context.rb:197:in `block in evaluate'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/context.rb:194:in `each'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/context.rb:194:in `evaluate'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/processed_asset.rb:12:in `initialize'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:374:in `new'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:374:in `block in build_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:395:in `circular_call_protection'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:373:in `build_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/index.rb:94:in `block in build_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/caching.rb:51:in `cache_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/index.rb:93:in `build_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:287:in `find_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/index.rb:61:in `find_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/processed_asset.rb:111:in `block in resolve_dependencies'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/processed_asset.rb:105:in `each'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/processed_asset.rb:105:in `resolve_dependencies'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/processed_asset.rb:97:in `build_required_assets'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/processed_asset.rb:16:in `initialize'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:374:in `new'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:374:in `block in build_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:395:in `circular_call_protection'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:373:in `build_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/index.rb:94:in `block in build_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/caching.rb:51:in `cache_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/index.rb:93:in `build_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:287:in `find_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/index.rb:61:in `find_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/bundled_asset.rb:16:in `initialize'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:377:in `new'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:377:in `build_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/index.rb:94:in `block in build_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/caching.rb:51:in `cache_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/index.rb:93:in `build_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:287:in `find_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/index.rb:61:in `find_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/environment.rb:75:in `find_asset'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/sprockets-2.10.0/lib/sprockets/base.rb:295:in `[]'
/home/mogztter/workspaces/asciidoctor.js/Rakefile:20:in `block in <top (required)>'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/bin/ruby_noexec_wrapper:14:in `eval'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/bin/ruby_noexec_wrapper:14:in `<main>'
Tasks: TOP => dist
(See full trace by running task with --trace)

Am I  doing something wrong ?

Cheers,
Guillaume.

Thanks Dan for the detailed explanation.
I picked some of your changes from https://github.com/mojavelinux/asciidoctor/commits/a51027d9f99f8d9c06b475e70595864f27413ffd

Do you know how can I rewrite this syntax in asciidoctor/lib/asciidoctor/document.rb:
@attribute_overrides = (options[:attributes] || {}).inject({}) do |collector,(key,value)|

The compiler complains :
rake aborted!
parse error on value "(" (PAREN_BEG) :asciidoctor/document:124
  (in /home/mogztter/workspaces/asciidoctor.js/asciidoctor/lib/asciidoctor/document.rb)
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/opal-0.4.4/lib/opal/lexer.rb:40:in `on_error'
(eval):3:in `_racc_do_parse_c'
(eval):3:in `do_parse'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/opal-0.4.4/lib/opal/lexer.rb:33:in `parse'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/opal-0.4.4/lib/opal/parser.rb:76:in `parse'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/opal-0.4.4/lib/opal/require_parser.rb:17:in `parse'
/home/mogztter/.rvm/gems/ruby-1.9.3-p429/gems/opal-sprockets-0.2.0/lib/opal/sprockets/processor.rb:73:in `evaluate'

By "upstream for 1.5.0" you mean 0.1.5 ? or are you planning to release Asciidoctor 1.0.0 anytime soon ? ;)
Can I start sending PR with my changes to Asciidoctor upstream ?

Thanks,
Guillaume.

A little bit verbose but it seems to be working :
@attribute_overrides = (options[:attributes] || {}).inject({}) do |collector,hash|
  key, value = hash

I copied the new html5 erb templates.

The templates below make the Opal compiler angy :
lib/asciidoctor/backends/html5/block_listing.opalerb
lib/asciidoctor/backends/html5/block_video.opalerb
lib/asciidoctor/backends/html5/inline_image.opalerb
lib/asciidoctor/backends/html5/section.opalerb

"backslash must appear before newline" [1]. Maybe someone is familiar with this error ?

I am now stuck with a Javascript error when running Asciidoctor.js :

Uncaught TypeError: Cannot call method '$[]' of undefined asciidoctor.js:2181
def.$lock_in_subs asciidoctor.js:2181
Lexer.constructor.$next_block asciidoctor.js:4352
Lexer.constructor.$next_section asciidoctor.js:3875
Lexer.constructor.$parse

default_subs = $scope.SUBS is undefined and the $scope is initialized with $scope = AbstractBlock._scope

The SUBS constants from Substituters are not loaded in the "scope" but I don't know why... ideas ?


Guillaume.
[1] https://github.com/opal/opal/blob/master/lib/opal/lexer.rb#L582

mojavelinux wrote

Is your 0.1.4 branch on GitHub the same as what you are compiling locally?
Let me know if / when they are synced up and I'll follow along on my side
to help you work through the issues.

Yes they are but the branch was really messy because I tried to go as far as I can before rewriting history.

I created an up-to-date branch named 1.5.0. The Asciidoctor submodule points at a branch named Issue#679 (created from Asciidoctor master just a minute ago).

The current error is : Uncaught NoMethodError: undefined method `home' for Dir

Let's make asciidoctor.js up to date !

Cheers,
Guillaume.

I have a strange behavior with a quite simple Ruby code:

RUBY_ENGINE = 'unknown' unless defined? RUBY_ENGINE
RUBY_ENGINE_OPAL = (RUBY_ENGINE == 'opal')
RUBY_ENGINE_JRUBY = (RUBY_ENGINE == 'jruby')

This code compiles in Javascript as :

  if (($a = ($scope.RUBY_ENGINE != null)) === false || $a === nil) {
    $scope.RUBY_ENGINE = "unknown"
  };
  $scope.RUBY_ENGINE_OPAL = $scope.RUBY_ENGINE['$==']("opal");
  $scope.RUBY_ENGINE_JRUBY = $scope.RUBY_ENGINE['$==']("jruby");

I added console.log because $scope.RUBY_ENGINE_OPAL was false :

  console.log("$scope.RUBY_ENGINE: '" + $scope.RUBY_ENGINE + "'");
  console.log("$scope.RUBY_ENGINE['$==']: " + $scope.RUBY_ENGINE['$==']);
  console.log("$scope.RUBY_ENGINE['$=='](\"opal\"): " + ($scope.RUBY_ENGINE['$==']("opal")));

Output :

$scope.RUBY_ENGINE: 'opal'
$scope.RUBY_ENGINE['$==']: function (other) {
      var self = this;
      return self === other;
    } 
$scope.RUBY_ENGINE['$==']("opal"): false 

Just to be sure, I modified the Javascript code to :

$scope.RUBY_ENGINE_OPAL = $scope.RUBY_ENGINE === "opal";

And its works, $scope.RUBY_ENGINE_OPAL is now true...
strange no ?

mojavelinux wrote

I'm not sure. Maybe something Opal doesn't support.

We could change the references to SUBS that fall outside of the
substutitiors class to an absolute reference. Something like
::Asciidoctor::Substituters::SUBS. See if that works.

Thanks that solved the problem. I pushed all my work to the 1.5.0 branch.

Excellent! I'll track that branch and send feedback once I've given it a try.

Great work so far. We're really making headway. It requires patience since success is binary in this case. It fails, it fails, it fails, it fails, then it works!

-Dan

Hi, I'm stuck with an error and I don't know how to handle this case. The warn method from the Kernel module is not found by Opal:

Uncaught NoMethodError: undefined method `warn' for Asciidoctor::Lexer

I tried to add "require 'kernel'" to solve this issue and it nearly worked... but then Opal mixed up definition of methods. For example the =~ method now use the implementation of the Kernel module (opal/corelib/kernel.rb) instead of the implementation of the String class or Regexp class :

module Kernel
# ...
  def =~(obj)
    false
  end
end
# ...

Any idea ? Maybe it's some kind of "bug" or limitation in Opal ?

To move on I commented out all usage of the warn method :/

Cheers,
Guillaume