asciidoctor and pygments not working under cygwin64

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

asciidoctor and pygments not working under cygwin64

BigBlackDog
This post was updated on .
I have cygwin64 running on my win10 machine. I've installed Ruby v2.2.5p319, Python 2.7.12 and the following gems:

asciidoctor (1.5.5)
coderay (1.1.1)
json (1.8.2)
minitest (5.4.3, 4.7.5)
multi_json (1.12.1)
power_assert (0.2.2)
pygments.rb (1.1.1)
rake (10.4.2)
rdoc (4.2.0)
test-unit (3.0.8)

pygemts ist running at version 2.0.2-1. (edited post)

i have this simple document to test the syntax-highlighting with pygments:

= Document Title
Doc Writer <doc@example.com>
:doctype: book
:reproducible:
:source-highlighter: pygments
:listing-caption: Listing

A simple http://asciidoc.org[AsciiDoc] document.

== just a code sample
.Create a basic PDF document using Prawn
[source,ruby]
----
require 'prawn'

Prawn::Document.generate 'example.pdf' do
  text 'Hello, World!'
end
----

now, when i try and generate a html file by running asciidoctor test.adoc ... i get no output and the task keeps running and running and running without generating any output.

when i change the highligher to coderay i get the desired output. but i like the pygments output more and want to use it.

It was running on an older machine with earlier versions of everything.

can someone help me figure out whats going wrong?

thank you!
Reply | Threaded
Open this post in threaded view
|

Re: asciidoctor and pygments not working under cygwin64

mojavelinux
Administrator
The pygments.rb recently underwent an internal change to support Ruby >= 2.4 and JRuby. It no longer uses posix/spawn to invoke Python. Now it just uses popen4. There's a chance this change has caused a problem on Windows. But I run the Jekyll AsciiDoc tests on AppVeyor, which include calls to Pygments, and it works fine.


If you continue to have a problem, you probably need to report this to the pygments.rb project since Asciidoctor is simply delegating to that library.


You might try switching to Ruby 2.3 or 2.4, though I'm not confident that's where the issue is.

The other thing you can try is to install pygments.rb 0.6.3. This version will work on Ruby < 2.4.

$ gem uninstall pygments.rb
  gem install pygments.rb --version 0.6.3

That should at least restore the old behavior.

Cheers,

-Dan

On Thu, Jan 19, 2017 at 3:15 AM, BigBlackDog [via Asciidoctor :: Discussion] <[hidden email]> wrote:
I have cygwin64 running on my win10 machine. I've installed Ruby v2.2.5p319, Python 2.7.12 and the following gems:

asciidoctor (1.5.5)
coderay (1.1.1)
json (1.8.2)
minitest (5.4.3, 4.7.5)
multi_json (1.12.1)
power_assert (0.2.2)
pygments.rb (1.1.1)
rake (10.4.2)
rdoc (4.2.0)
test-unit (3.0.8)

i have this simple document to test the syntax-highlighting with pygments:

= Document Title
Doc Writer <[hidden email]>
:doctype: book
:reproducible:
:source-highlighter: pygments
:listing-caption: Listing

A simple http://asciidoc.org[AsciiDoc] document.

== just a code sample
.Create a basic PDF document using Prawn
[source,ruby]
----
require 'prawn'

Prawn::Document.generate 'example.pdf' do
  text 'Hello, World!'
end
----

now, when i try and generate a html file by running asciidoctor test.adoc ... i get no output and the task keeps running and running and running without generating any output.

when i change the highligher to coderay i get the desired output. but i like the pygments output more and want to use it.

It was running on an older machine with earlier versions of everything.

can someone help me figure out whats going wrong?

thank you!


If you reply to this email, your message will be added to the discussion below:
http://discuss.asciidoctor.org/asciidoctor-and-pygments-not-working-under-cygwin64-tp5212.html
To start a new topic under Asciidoctor :: Discussion, email [hidden email]
To unsubscribe from Asciidoctor :: Discussion, click here.
NAML



--
Dan Allen | @mojavelinux | https://twitter.com/mojavelinux
Reply | Threaded
Open this post in threaded view
|

Re: asciidoctor and pygments not working under cygwin64

BigBlackDog
Thank you so much for your support!

Downgrading to pygments 0.6.3 did help me to get to the desired output.

I'll file a report there to help getting current versions work together.
Reply | Threaded
Open this post in threaded view
|

Re: asciidoctor and pygments not working under cygwin64

wolandscat
In reply to this post by mojavelinux
Dan,
I just happen to have upgraded some asciidoctor and related gems under cygwin64 under Windows 10, and hit exactly the same problem as described in the first post above. I am on Ruby 2.6.4 and pygments.rb 1.2.1. Running asciidoctor on any doc with syntax in it causes a hang.

But if I now try to install pygments.rb 0.6.3, it says there is no pygments.rb of that version anywhere.

The last code commit I see in pygments.rb on Github is end 2017, so it looks unmaintained.

I used pygments.rb with asciidoctor because I wrote a couple of modes for it for a couple of syntaxes we use in our standards, so moving to another syntax processor isn't that desirable (although if we get desperate...).

Right now my asciidoctor publishing env is broken, and I can't seem to fix it...

Any thoughts?
Reply | Threaded
Open this post in threaded view
|

Re: asciidoctor and pygments not working under cygwin64

mojavelinux
Administrator
To be honest, I'm not that familiar with the cygwin environment, so I'm not really sure what to tell you. We do test Asciidoctor and Pygments together on Windows for each commit. So I know that combination works fine.

> But if I now try to install pygments.rb 0.6.3, it says there is no pygments.rb of that version anywhere.

The version of the gem is still on rubygems.org (https://rubygems.org/gems/pygments.rb/versions/0.6.3), so I'm not sure why you are getting that error.

> The last code commit I see in pygments.rb on Github is end 2017, so it looks unmaintained.

GitHub itself no longer uses Pygments for source highlighting, and since they originally started the Ruby bindings (pygments.rb), I don't expect that project is really going to go anywhere. Rouge stepped in as its replacement and has since become the recommended syntax highlighter in Ruby. It receives quite a bit of activity, esp over the past year. https://github.com/rouge-ruby/rouge

> I used pygments.rb with asciidoctor because I wrote a couple of modes for it for a couple of syntaxes we use in our standards, so moving to another syntax processor isn't that desirable (although if we get desperate...).

You're free to continue to use pygments.rb, whether it's 0.6.3 or 1.2.1. But we can't control what happens with the syntax highlighter libraries. They are going to change, and that sometimes means making a migration. Although there should be no immediate need, I would recommend starting to explore the switch from Pygments to Rouge.

Best,

-Dan

On Tue, Dec 10, 2019 at 5:09 AM wolandscat [via Asciidoctor :: Discussion] <[hidden email]> wrote:
Dan,
I just happen to have upgraded some asciidoctor and related gems under cygwin64 under Windows 10, and hit exactly the same problem as described in the first post above. I am on Ruby 2.6.4 and pygments.rb 1.2.1. Running asciidoctor on any doc with syntax in it causes a hang.

But if I now try to install pygments.rb 0.6.3, it says there is no pygments.rb of that version anywhere.

The last code commit I see in pygments.rb on Github is end 2017, so it looks unmaintained.

I used pygments.rb with asciidoctor because I wrote a couple of modes for it for a couple of syntaxes we use in our standards, so moving to another syntax processor isn't that desirable (although if we get desperate...).

Right now my asciidoctor publishing env is broken, and I can't seem to fix it...

Any thoughts?



If you reply to this email, your message will be added to the discussion below:
https://discuss.asciidoctor.org/asciidoctor-and-pygments-not-working-under-cygwin64-tp5212p7404.html
To start a new topic under Asciidoctor :: Discussion, email [hidden email]
To unsubscribe from Asciidoctor :: Discussion, click here.
NAML


--
Dan Allen | @mojavelinux | https://twitter.com/mojavelinux
Reply | Threaded
Open this post in threaded view
|

Re: asciidoctor and pygments not working under cygwin64

wolandscat
I have had a look at the Rouge approach. It appears that this involves writing lexers as Ruby source files. I'm not personally against setting up a Ruby environment and learning enough Ruby to make that work, but it is not a very generic approach.

A better way would be to allow the use of something like Antlr4 files or some other independent lexer specification file that can be re-used across any particular implementation - e.g. how gvim does it, and how most parser builders work.

Ideally, one would be able to write a universally re-usable lexer file for any language that any syntax highlighting system, as well as other tools could use.

Aside: cygwin is much closer to a posix environment than Windows; everything runs inside bash, so testing on Windows native won't reveal anything about what might happen in cygwin. Generally speaking, if it works on Linux it works on cygwin. Which is why this anomaly is a surprise.
Reply | Threaded
Open this post in threaded view
|

Re: asciidoctor and pygments not working under cygwin64

mojavelinux
Administrator
If something that works on Linux works on cygwin without any issues, then I expect it to work. That means you should not have start writing lexers immediately. I'm not really sure what else to tell you, other than check your environment.

> A better way would be to allow the use of something like Antlr4 files or some other independent lexer specification file that can be re-used across any particular implementation - e.g. how gvim does it, and how most parser builders work.

That's certainly a good idea. I've voiced my opinion in the past about the lack of portability with grammars for syntax highlighters. But we're not in the business of making syntax highlighters here, so you'd have direct your feedback at those projects. Asciidoctor is merely integrating what's out there.

Best,

-Dan

On Mon, Dec 30, 2019 at 11:58 AM wolandscat [via Asciidoctor :: Discussion] <[hidden email]> wrote:
I have had a look at the Rouge approach. It appears that this involves writing lexers as Ruby source files. I'm not personally against setting up a Ruby environment and learning enough Ruby to make that work, but it is not a very generic approach.

A better way would be to allow the use of something like Antlr4 files or some other independent lexer specification file that can be re-used across any particular implementation - e.g. how gvim does it, and how most parser builders work.

Ideally, one would be able to write a universally re-usable lexer file for any language that any syntax highlighting system, as well as other tools could use.

Aside: cygwin is much closer to a posix environment than Windows; everything runs inside bash, so testing on Windows native won't reveal anything about what might happen in cygwin. Generally speaking, if it works on Linux it works on cygwin. Which is why this anomaly is a surprise.


If you reply to this email, your message will be added to the discussion below:
https://discuss.asciidoctor.org/asciidoctor-and-pygments-not-working-under-cygwin64-tp5212p7462.html
To start a new topic under Asciidoctor :: Discussion, email [hidden email]
To unsubscribe from Asciidoctor :: Discussion, click here.
NAML


--
Dan Allen | @mojavelinux | https://twitter.com/mojavelinux
Reply | Threaded
Open this post in threaded view
|

Re: asciidoctor and pygments not working under cygwin64

wolandscat
mojavelinux wrote
If something that works on Linux works on cygwin without any issues, then I
expect it to work. That means you should not have start writing lexers
immediately. I'm not really sure what else to tell you, other than check
your environment.
Sure - but it doesn't. This is not an AD problem of course, it's some sort of cygwin weirdness. The fact that it's been reported elsewhere means it's probably real. Probably I just need to accelerate the move of my main machine to Linux and get out of Windows/cygwin for good ;)

mojavelinux wrote
> A better way would be to allow the use of something like Antlr4 files or
some other independent lexer specification file that can be re-used across
any particular implementation - e.g. how gvim does it, and how most parser
builders work.

That's certainly a good idea. I've voiced my opinion in the past about the
lack of portability with grammars for syntax highlighters. But we're not in
the business of making syntax highlighters here, so you'd have direct your
feedback at those projects. Asciidoctor is merely integrating what's out
there.
Agree - I'll try and track down where/how to do that. Nevertheless, I think that if the core AD developer group were to suggest to the Rouge developers a separation of lexer files from the Ruby source code environment into Antlr4 or whatever other format consumable by the binary tool env, it might have more effect than just a random individual such as myself making such a suggestion.

thanks

- thomas