Login  Register

Memory settings for developing AsciidoctorJ in IntelliJ IDEA

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options Options
Embed post
Permalink
Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Memory settings for developing AsciidoctorJ in IntelliJ IDEA

mojavelinux
Administrator
2681 posts
Several people, including myself, have reported problems developing AsciidoctorJ using IntelliJ IDEA since the switch to Gradle. I'd like to report what I discovered and propose a few solutions.

Fortunately, IntelliJ IDEA 14 provides Gradle support out of the box, so you can simply import the project by pointing IntelliJ IDEA at the root build.gradle file. However, one the project is imported, that's when the problems begin.

I've been observing two problems:

1. OutOfMemory error: unable to create new native thread (doing almost anything)
2. Timeout trying to debug a test

First, I've determined that IDEA seems to be happiest running the Oracle JDK. I have switched all my IDEA settings to Oracle JDK 1.7.0. That includes the settings for launching IDEA and the settings for the Project SDK. Even if you aren't using Oracle JDK, make sure you are using the same JDK everywhere.

Second, I found that when I switched to launching with Oracle JDK (on Linux), I lost font antialiasing. I reenabled by adding the following line to idea64.vmoptions:

 -Dawt.useSystemAAFontSettings=gasp

This value was previously set to "lcd".

Next, I did some more tweaking of memory settings in that file to give more headroom for the JVM. Here are the option I settled with. Your milage may vary, but it's a starting point:

....
-server
-Xms512m
-Xmx2048m
-XX:PermSize=256m
-XX:MaxPermSize=512m
-XX:ReservedCodeCacheSize=256m
-ea
-Dsun.io.useCanonCaches=false
-Djava.net.preferIPv4Stack=true
-Djsse.enableSNIExtension=false
-XX:+UseCodeCacheFlushing
-XX:+UseConcMarkSweepGC
-XX:SoftRefLRUPolicyMSPerMB=50
-Dawt.useSystemAAFontSettings=gasp
-XX:+CMSClassUnloadingEnabled
-XX:+TieredCompilation
....

The -XX:+TieredCompilation flag is going to help since we are working with JRuby.

Having made all those changes, I was *still* getting OutOfMemory errors. After a few restarts, I determined that it was happening whenever I had Chrome running. I know that Chrome uses a lot of processes, and the Gradle daemon requires yet another. That led me to process limits. By default, the max number of processes on a Linux box is typically 1024. I increased it 4 fold:

 $ ulimit -u 4096

Once I made this change, my OutOfMemory problems went away. (Add it to ~/.bash_profile to make it permanent).

Finally, I went into the Gradle settings in IDEA and added the following to the VM options field to give the daemon more headroom as well:

 -Xmx2048m -XX:MaxPermSize=512m

I'll know more in a couple of days, but it seems like the combination of these changes has gotten rid of the memory problems when developing AsciidoctorJ in IntelliJ IDEA. If you have different results or experiences, please feel free to share!

Cheers,

Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Re: Memory settings for developing AsciidoctorJ in IntelliJ IDEA

LightGuardjp
354 posts
What about running on Java 8? Did that change things much?

On Thursday, December 18, 2014, mojavelinux [via Asciidoctor :: Discussion] <[hidden email]> wrote:
Several people, including myself, have reported problems developing AsciidoctorJ using IntelliJ IDEA since the switch to Gradle. I'd like to report what I discovered and propose a few solutions.

Fortunately, IntelliJ IDEA 14 provides Gradle support out of the box, so you can simply import the project by pointing IntelliJ IDEA at the root build.gradle file. However, one the project is imported, that's when the problems begin.

I've been observing two problems:

1. OutOfMemory error: unable to create new native thread (doing almost anything)
2. Timeout trying to debug a test

First, I've determined that IDEA seems to be happiest running the Oracle JDK. I have switched all my IDEA settings to Oracle JDK 1.7.0. That includes the settings for launching IDEA and the settings for the Project SDK. Even if you aren't using Oracle JDK, make sure you are using the same JDK everywhere.

Second, I found that when I switched to launching with Oracle JDK (on Linux), I lost font antialiasing. I reenabled by adding the following line to idea64.vmoptions:

 -Dawt.useSystemAAFontSettings=gasp

This value was previously set to "lcd".

Next, I did some more tweaking of memory settings in that file to give more headroom for the JVM. Here are the option I settled with. Your milage may vary, but it's a starting point:

....
-server
-Xms512m
-Xmx2048m
-XX:PermSize=256m
-XX:MaxPermSize=512m
-XX:ReservedCodeCacheSize=256m
-ea
-Dsun.io.useCanonCaches=false
-Djava.net.preferIPv4Stack=true
-Djsse.enableSNIExtension=false
-XX:+UseCodeCacheFlushing
-XX:+UseConcMarkSweepGC
-XX:SoftRefLRUPolicyMSPerMB=50
-Dawt.useSystemAAFontSettings=gasp
-XX:+CMSClassUnloadingEnabled
-XX:+TieredCompilation
....

The -XX:+TieredCompilation flag is going to help since we are working with JRuby.

Having made all those changes, I was *still* getting OutOfMemory errors. After a few restarts, I determined that it was happening whenever I had Chrome running. I know that Chrome uses a lot of processes, and the Gradle daemon requires yet another. That led me to process limits. By default, the max number of processes on a Linux box is typically 1024. I increased it 4 fold:

 $ ulimit -u 4096

Once I made this change, my OutOfMemory problems went away. (Add it to ~/.bash_profile to make it permanent).

Finally, I went into the Gradle settings in IDEA and added the following to the VM options field to give the daemon more headroom as well:

 -Xmx2048m -XX:MaxPermSize=512m

I'll know more in a couple of days, but it seems like the combination of these changes has gotten rid of the memory problems when developing AsciidoctorJ in IntelliJ IDEA. If you have different results or experiences, please feel free to share!

Cheers,




If you reply to this email, your message will be added to the discussion below:
http://discuss.asciidoctor.org/Memory-settings-for-developing-AsciidoctorJ-in-IntelliJ-IDEA-tp2572.html
To start a new topic under Asciidoctor :: Discussion, email <a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;ml-node%2Bs49171n1h37@n6.nabble.com&#39;);" target="_blank">ml-node+s49171n1h37@...
To unsubscribe from Asciidoctor :: Discussion, click here.
NAML


--

Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Re: Memory settings for developing AsciidoctorJ in IntelliJ IDEA

mojavelinux
Administrator
2681 posts

Nope. I switched to 1.7.0 since it doesn't seem like IntelliJ gets tested much on 1.8.0.

I will try now that I figured out the ulimit problem.

-Dan

Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Re: Memory settings for developing AsciidoctorJ in IntelliJ IDEA

Robert.Panzer
70 posts
I can only contribute new problems, no solutions:(

I did not see memory problems on OSX or Windows 7.

On OSX building works fine.
But debugging via gradle does not work. (Unfortunately I cannot share the error message as I am sitting in front of a Windows box.)
Gradle seems to start the tests with JDWP as a TCP client and a port that changes with every start.
And this port does not match the server port that is configured in Settings->Build, Execution, Deployment->Debugger->Built-in server port.

Workaround is to directly execute as a JUnit test without using gradle.
(Right-click the test class, select Run or Debug in the context menu. The next menu that opens contains two items, one with a Gradle symbol, one with another. Select the latter.)

On Windows building AsciidoctorJ is not possible at all due to https://github.com/jruby-gradle/jruby-gradle-plugin/issues/83
Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Re: Memory settings for developing AsciidoctorJ in IntelliJ IDEA

mojavelinux
Administrator
2681 posts

On Thu, Dec 18, 2014 at 10:57 PM, Robert.Panzer [via Asciidoctor :: Discussion] <[hidden email]> wrote:
Gradle seems to start the tests with JDWP as a TCP client and a port that changes with every start.

I've observed the same. 
 
And this port does not match the server port that is configured in Settings->Build, Execution, Deployment->Debugger->Built-in server port.

Here's some info about how to tune it:

 

Workaround is to directly execute as a JUnit test without using gradle.

I think this is the best approach for debugging. There's really no reason to involve Gradle for this task.

Perhaps we can document this in the (yet to be created) CONTRIBUTING-CODE.adoc in the repository.

-Dan

--
Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Re: Memory settings for developing AsciidoctorJ in IntelliJ IDEA

asotobu
298 posts
Cool, I get the same exceptions as Robert. It seems that Linux + IntelliJ + Gradle memory problems. MacOsX + Intellij + Gradle Debug problems hehehe
Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Re: Memory settings for developing AsciidoctorJ in IntelliJ IDEA

Robert.Panzer
70 posts
In reply to this post by mojavelinux
Thanks for the link!
The gradle forum discussion rather discusses how to tell gradle that an IDE can connect to the test VM.
But IntelliJ seems to configure gradle so that it expects the test VM to connect to the IDE and therefore the test VM is started with an option like this:

    -Xrunjdwp:transport=dt_socket,server=n,suspend=?,address=<ephemeral port>

I will investigate again over the weekend, probably IntelliJ sets the wrong GRADLE_OPTS.

While I agree that for most of the times a direct execution is the better way because it is much faster, I think that the other way around should work too to get the whole build pipeline executed instead of having to build and execute.
And there may be build files where build.gradle sets a complex set of properties for the test that I don't want to reconfigure in the IntelliJ run configuration.

(And certainly it should simply work, the rest of the gradle integration is great.)
Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Re: Memory settings for developing AsciidoctorJ in IntelliJ IDEA

mojavelinux
Administrator
2681 posts
Keep in mind there is a dialog in the settings where you can pass vm options to Gradle. You could see if it's possible to override these options using that dialog (though it may not work).

On Fri, Dec 19, 2014 at 1:49 AM, Robert.Panzer [via Asciidoctor :: Discussion] <[hidden email]> wrote:
While I agree that for most of the times a direct execution is the better way because it is much faster, I think that the other way around should work too to get the whole build pipeline executed instead of having to build and execute.

+1

Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Re: Memory settings for developing AsciidoctorJ in IntelliJ IDEA

mojavelinux
Administrator
2681 posts
In reply to this post by Robert.Panzer

On Thu, Dec 18, 2014 at 10:57 PM, Robert.Panzer [via Asciidoctor :: Discussion] <[hidden email]> wrote:
On Windows building AsciidoctorJ is not possible at all due to https://github.com/jruby-gradle/jruby-gradle-plugin/issues/83

It looks like we have this problem resolved (in more ways than one).

The AsciidoctorJ build is now passing on Windows. Can you confirm once you get a chance?

https://ci.appveyor.com/project/asciidoctor/asciidoctorj

-Dan
Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Re: Memory settings for developing AsciidoctorJ in IntelliJ IDEA

Robert.Panzer
70 posts
Sure! Will do on Monday morning!



Am 21.12.2014 um 08:04 schrieb mojavelinux [via Asciidoctor :: Discussion] <[hidden email]>:


On Thu, Dec 18, 2014 at 10:57 PM, Robert.Panzer [via Asciidoctor :: Discussion] <[hidden email]> wrote:
On Windows building AsciidoctorJ is not possible at all due to https://github.com/jruby-gradle/jruby-gradle-plugin/issues/83

It looks like we have this problem resolved (in more ways than one).

The AsciidoctorJ build is now passing on Windows. Can you confirm once you get a chance?

https://ci.appveyor.com/project/asciidoctor/asciidoctorj

-Dan



If you reply to this email, your message will be added to the discussion below:
http://discuss.asciidoctor.org/Memory-settings-for-developing-AsciidoctorJ-in-IntelliJ-IDEA-tp2572p2596.html
To unsubscribe from Memory settings for developing AsciidoctorJ in IntelliJ IDEA, click here.
NAML
Reply | Threaded
Open this post in threaded view
| More
Print post
Permalink

Re: Memory settings for developing AsciidoctorJ in IntelliJ IDEA

Robert.Panzer
70 posts
Just to have it in this forum as well:

The AsciidoctorJ build or the jrubyPrepareGems task respectively succeeds with the current snapshot of the ruby-gradle-plugin.

I used commit a3c5bd0536f05b06603e03cc808abba84780be73 from https://github.com/jruby-gradle/jruby-gradle-plugin.

To use it in the asciidoctorJ build I had to add the plugin I compiled myself to the buildscript dependencies and remove the official plugin:


    // legacy plugins config; necessary to accomodate Java 6
    buildscript {
      repositories {
        jcenter()
      }
      dependencies {
        if (JavaVersion.current().isJava7Compatible()) {
          classpath 'com.netflix.nebula:nebula-publishing-plugin:2.0.0'
        }
        classpath files('C:/work/git/jruby-gradle-plugin/build/libs/jruby-gradle-plugin-0.1.10-SNAPSHOT.jar')
      }
    }
   
    // modern plugins config
    plugins {
      //id 'com.github.jruby-gradle.base' version '0.1.10'
      // Adding the nebula-publishing plugin when using JDK6 causes the build to abort
      //id 'nebula.nebula-publishing' version '2.0.0'
      id 'com.jfrog.bintray' version '1.0'
    }

Note: jruby-gradle-plugin 0.1.10-SNAPSHOT is actually a misspelling, should be 0.1.11-SNAPSHOT as 0.1.10 is already released. But its build file still names it 0.1.10.