Every supported diagram type in Asciidoctor Diagram is itself an extension of Asciidoctor and depends on a common DiagramConverter provided by Asciidoctor Diagram.
You should be able to add a new diagram type by following the example of for example PlantUML:
https://github.com/asciidoctor/asciidoctor-diagram/blob/master/lib/asciidoctor-diagram/plantuml.rb and
https://github.com/asciidoctor/asciidoctor-diagram/tree/master/lib/asciidoctor-diagram/plantumlAs adding new diagram types depend on Asciidoctor Diagram Ruby code, I doubt that you can write add a diagram type by using just Java code although I might be mistaken here.
Regards,
Alexander