How to generate fonts

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

How to generate fonts

chloerei
This post was updated on .
I'm using asciidoctor-pdf, and I want to render a document write in CJK charsets, so I need to add a CJK font in theme.

I have read https://github.com/asciidoctor/asciidoctor-epub3/blob/master/scripts/generate-font-subsets.pe , it looks complex than I think.

I don't need subsets, so I use this script to convert Noto CJK font from OTF to TTF:

[source]
----
#!/usr/bin/env fontforge
Open($1)
Generate($1:r + ".ttf")
----

But it drops the CJK, how to convert all fonts?

And I found I must convert four styles(normal, bold, italic, bold_italic) for each fonts. Could you release a more simple script to convert a font to four styles without subsets? It works like:

[source]
----
$ generate-font NotoSansCJKsc-Regular.otf
Genrating...
NotoSansCJKsc-Regular-Regular.ttf
NotoSansCJKsc-Regular-Bold.ttf
NotoSansCJKsc-Regular-Italic.ttf
NotoSansCJKsc-Regular-BoldItalic.ttf
----

This example look duplicate with regular and bold, because Noto CJK has release different weight, but it's useful for some font only have one weight(like DroidFallback).

I don't konw font format's detail, so I probably misunderstood, it it possible?
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

mojavelinux
Administrator
I'll follow up with some detail as soon as I get a chance.

Until then, I highly recommend the following two (free) resources.


That's how I learned enough to create the existing script.

As a quick hint, you'll want to select all the characters using something like:

SelectMore(0u0020,0u1eff)

But I recommend reading through at least the scripting tutorial. At first, it's pretty dense, but once you get a handle on a few concepts, you'll find tremendous power in being able to work with fonts, including merging and remapping them :)

Cheers,

-Dan


On Mon, Oct 20, 2014 at 11:52 PM, chloerei [via Asciidoctor :: Discussion] <[hidden email]> wrote:
I'm using asciidoctor-pdf, and I want to render a document write in CJK charsets, so I need to add a CJK font in theme.

I have read https://github.com/asciidoctor/asciidoctor-epub3/blob/master/scripts/generate-font-subsets.pe , it looks complex than I think.

I don't need subsets, so I use this script to covert Noto CJK font from OTF to TTF:

[source]
----
#!/usr/bin/env fontforge
Open($1)
Generate($1:r + ".ttf")
----

But it drops the CJK, how to convert all fonts?

And I found I must convert four styles(normal, bold, italic, bold_italic) for each fonts. Could you release a more simple script to convert a font to four styles without subsets? It works like:

[source]
----
$ generate-font NotoSansCJKsc-Regular.otf
Genrating...
NotoSansCJKsc-Regular-Regular.ttf
NotoSansCJKsc-Regular-Bold.ttf
NotoSansCJKsc-Regular-Italic.ttf
NotoSansCJKsc-Regular-BoldItalic.ttf
----

This example look duplicate with regular and bold, because Noto CJK has release different weight, but it's useful for some font only have one weight(like DroidFallback).

I don't konw font format's detail, so I probably misunderstood, it it possible?


If you reply to this email, your message will be added to the discussion below:
http://discuss.asciidoctor.org/How-to-generate-fonts-tp2358.html
To start a new topic under Asciidoctor :: Discussion, email [hidden email]
To unsubscribe from Asciidoctor :: Discussion, click here.
NAML



--
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

chloerei
This post was updated on .
In reply to this post by chloerei
Share some experience after one day learning.

Noto CJK use CID, so it need flatten before generate TTF:

[source]
----
#!/usr/bin/env fontforge
Open($1)
CIDFlatten()
Generate($1:r + ".ttf")
----

And if want to generate italic style:

[source]
----
#!/usr/bin/env fontforge
Open($1)
CIDFlatten()
SelectAll()
Skew(10)
Generate($1:r + "-italic.ttf")
----

But I still found some issues, for example can not display blank space, maybe my script break something.

This time I give up convert Noto CJK OTF to TTF, just use DroidSansFallbackFull.ttf for all style, and back to my primary task -- writing. Hope some one will share a better way to use custom font in the future.
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

mojavelinux
Administrator

On Wed, Oct 22, 2014 at 2:04 AM, chloerei [via Asciidoctor :: Discussion] <[hidden email]> wrote:
This time I give up convert Noto CJK OTF to TTF, just use DroidSansFallbackFull.ttf for all style, and back to my primary task -- writing. Hope some one will share a better way to use custom font in the future.

My plan is definitely to provide CJK support for the Noto font in Asciidoctor EPUB3 and PDF, so I'll definitely get to fixing up this script in the near future. I did quite a bit of research about how to use FontForge, so I'm fairly confident I can make it happen. Of course, I'll need validation that I did it right ;) I'll be in touch as soon as I can get to it.

Cheers,

Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

chloerei
I'm happy about that plan, and I'm willing to join the test. I think Asciidoctor will be the best publishing toolchain for technology author.
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

mojavelinux
Administrator
Rei,

This one turned out to be more complicated than I expected. The first problem is that the CJK font in the Noto family is NotoSans, not NotoSerif. That sort of makes sense given the nature of the glyphs, but it makes it so that we have to introduce a separate theme for CJK documents that uses NotoSans as the primary font.

I worked out the fontforge script needed to take the NotoSansCJKsc OTF file and make it something that Asciidoctor PDF or Asciidoctor EPUB3 can use.

https://gist.github.com/mojavelinux/c74162d44ca77d1a7c25

You were having a problem with the space character because, oddly, NotoSansCJKsc is missing the space character (0x0020). The following code from the script recreates the missing glyph by stealing content from the non-break space.

----
Select(0u00a0)
Copy()
Select(0u0020)
Paste()
SetWidth(0)
SelectNone()
----

I ran the script in the gist on the NotoSansCJKsc otf file as follows:

 $ ./generate-cjk-font.pe NotoSansCJKsc-Regular.otf

I then renamed the TTF file:

 $ mv NotoSansCJKsc-Regular.otf notosans-regular-multilingual.ttf

(I didn't create the bold and italic fonts, but you'd follow a similar process)

After that, I needed to create a custom theme for PDF to use this font. I did that by copying the default-theme.yml file and changing the NotoSerif fonts to NotoSans. After that, I'm happy to say it worked!

I have to do some more thinking and hacking before I'm ready to add CJK support into Asciidoctor PDF. I'm probably not going to choose the Noto font family because the fonts are simply too large*. Instead, I'm likely going to use the M+ family. It will work either by specifying a CJK theme or by setting the `scripts` attribute in AsciiDoc to `cjk` or `multilingual`.

-Dan

* We may still have a separate gem to hold the cjk or multilingual theme to to avoid increasing the size of the gem for people not using cjk fonts.
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

chloerei
Thanks for your works.

I tried this script, and use the result font in project, but I found some characters are missing:

[source, asciidoc]
----
=== 环境变量

现在来了解一下什么是环境变量,以及如何设置环境变量。

环境变量是进程启动的时候由操作系统设置的一组变量。操作系统在启动的时候已经定义了一些环境变量,可以用 `env` 命令来查阅:
----

Compare DroidSansFallbackFull.ttf and DroidSansFallbackFull.ttf result





I think convert otf font to ttf maybe no a good idea, it bring many issuses, unless understand the whole font's detail, but it take too much time.

DroidSansFallbackFull.ttf is using in Ubuntu and old versions Android for cjk, it look acceptable in my document.(it missing some uncommon charsets, so Noto born). The perfect solution is directly support otf font in prawn, than editor can use the best font for his/her document, but I don't know how to write a otf parse for prawn.

I search M+ font (http://mplus-fonts.sourceforge.jp/), I found English and Japaness support, no Chinese and Korea, am I missing something?
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

mojavelinux
Administrator
Rei,

> I found some characters are missing:

If there are characters missing, just take out the following line from the fontforge script:

----
# select lesser range of glyphs
SelectMore(0u0020,0uffee)
SelectInvert()
Clear()
SelectNone()
----

> I think convert otf font to ttf maybe no a good idea

I think the problem is just that we shouldn't cut characters from the font until we know what we are cutting. I think the conversion from otf is working fine (well, there might be some side effects, but it isn't causing the missing glyphs).

These lines are cutting characters out of the font in an attempt to reduce the file size a bit. It's not necessary to do so.

> DroidSansFallbackFull.ttf is using in Ubuntu and old versions Android for cjk, it look acceptable in my document.

Yes, this is safe and indeed was my intention to have this supported in Asciidoctor PDF since PDF requires that you specify a font for every character (unlike a web browser). I saw your pull request, so I'll test that and merge it in. Thanks!

> I found English and Japaness support (in the M+ font), no Chinese and Korea, am I missing something?

I think the website is outdated. If you clone the repository, then you should see that you get all the characters. They are also packaged for Fedora and Ubuntu, so you can get them that way (see http://packages.ubuntu.com/trusty/fonts-mplus).

> The perfect solution is directly support otf font in prawn, than editor can use the best font for his/her document, but I don't know how to write a otf parse for prawn.

Neither do I and I'm not going to count on it happening any time soon. The best solution is to get a good set of TTF fonts that we can rely on and make sure they are well integrated in Asciidoctor PDF.

Cheers,

-Dan


On Thu, Jan 1, 2015 at 2:12 AM, chloerei [via Asciidoctor :: Discussion] <[hidden email]> wrote:
Thanks for your works.

I tried this script, and use the result font in project, but I found some characters are missing:

[source, asciidoc]
----
=== 环境变量

现在来了解一下什么是环境变量,以及如何设置环境变量。

环境变量是进程启动的时候由操作系统设置的一组变量。操作系统在启动的时候已经定义了一些环境变量,可以用 `env` 命令来查阅:
----

Compare DroidSansFallbackFull.ttf and DroidSansFallbackFull.ttf result





I think convert otf font to ttf maybe no a good idea, it bring many issuses, unless understand the whole font's detail, but it take too much time.

DroidSansFallbackFull.ttf is using in Ubuntu and old versions Android for cjk, it look acceptable in my document.(it missing some uncommon charsets, so Noto born). The perfect solution is directly support otf font in prawn, than editor can use the best font for his/her document, but I don't know how to write a otf parse for prawn.

I search M+ font (http://mplus-fonts.sourceforge.jp/), I found English and Japaness support, no Chinese and Korea, am I missing something?


If you reply to this email, your message will be added to the discussion below:
http://discuss.asciidoctor.org/How-to-generate-fonts-tp2358p2626.html
To start a new topic under Asciidoctor :: Discussion, email [hidden email]
To unsubscribe from Asciidoctor :: Discussion, click here.
NAML



--
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

chloerei
I remove `# select lesser range of glyphs` section and try again, ttf font size from 10.3mb to 17.0mb, but still missing charsets. ( like before, missing '什' and '量' )

> The best solution is to get a good set of TTF fonts that we can rely on and make sure they are well integrated in Asciidoctor PDF.

OK.
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

mojavelinux
Administrator

Perhaps these characters are just missing from the Noto Sans font? If you provide a list of several additional missing characters, that will help us debug.

It does seem that when I open the otf, there are a number of characters not assigned to a Unicode number. Perhaps this is the problem.

-Dan

Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

mojavelinux
Administrator
In reply to this post by chloerei
Aha! It appears we are bumping into bugs (or at least compatibility issues) left and right with these fonts, font format and fontforge.

There's a bug report specifically about this missing character when viewed in FontForge.

https://github.com/fontforge/fontforge/issues/1831

When I open the OTF font in FontForge v20120731, I see that there is no unicode number associated with 量 (0x91cf) {1}. However, I do see the character there. When we convert to TTF, it is indeed dropping this character because of the missing number (probably the purpose of the cid map).

It looks like the FontForge project has been rebooted {2}, so I grabbed the spec file and rebuilt the RPM (actually, I used the one from Fedora Rawhide). When I use FontForge v20140813, I see a unicode number associated at the location of the 量 glyph. However, the glyph doesn't look right...then again I don't know what it should look like and how much variation it can have. Perhaps you can try the latest FontForge, open the OTF, navigate to 0x91cf (Ctrl+Shift+> to bring up navigation dialog), and see if it looks acceptable to you.

My general conclusion is that we are hitting the bleeding edge with the Noto Sans CJK fonts and it might be a better strategy just to use CJK fonts that have working TTF files until the situation stabilizes. You have suggested using Droid Sans, and as I mentioned M+ is also an option. I'm partial to M+. I verified it does have the 量 glyph.

On another topic, it seems like the strategy for bold and italic in CJK is to use different font weights. Is that accurate? If so, then we should probably map the bold and italic to the font weights provided with these fonts (M+ for example has 7 weights).

Indeed, lots to learn. That's why it's important to document as we go.

Cheers,

-Dan

:1: http://www.unicode.org/cgi-bin/GetUnihanData.pl?codepoint=91CF
:2: https://github.com/fontforge/fontforge


On Thu, Jan 1, 2015 at 12:11 PM, Dan Allen <[hidden email]> wrote:

Perhaps these characters are just missing from the Noto Sans font? If you provide a list of several additional missing characters, that will help us debug.

It does seem that when I open the otf, there are a number of characters not assigned to a Unicode number. Perhaps this is the problem.

-Dan




--
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

chloerei
This post was updated on .
I tested FontForge v20141230 installed from Ubuntu PPA.

I checked location 0x91cf in NotoSans CJK, it's not the right '量' glyph. And checked same location in DroidSansFallback, it's right glyph.

But when I open NotoSans CJK in KDE's font viewer, enter example text '变量'(means 'variable'), it shows the right glyph. I don't understand the mechanism.

Than I checked mplus1p-regular-multilingual.ttf, I found a problem: it contains '量', but missing '变'(0x53d8), it's a common characters too. And I use it to generate PDF for my document, it missing a lot of characters. I think M+ have not enough feedback for Chinese, maybe it's good for Japaness and Korea.

Chinese has a lot of characters, common characters may have several thousand, when contains uncommon characters it comes to several ten thousand, and different regions have different variants (http://blog.typekit.com/2014/07/15/introducing-source-han-sans/). So CJK fonts will be large. NotoSans CJK is the most complete CJK font I know (different regions still need to install different font packages, because all CJK  characters up to 65535 limit. http://www.google.com/get/noto/cjk.html), DroidSansFallback is enough for common CJK use, M+ maybe not enough for Chinese common use.

Thanks for pay attention for CJK, I hope this content will be helpful.
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

chloerei
In reply to this post by mojavelinux
> On another topic, it seems like the strategy for bold and italic in CJK is to use different font weights. Is that accurate? If so, then we should probably map the bold and italic to the font weights provided with these fonts (M+ for example has 7 weights).

Font weight for bold, skew for italic.
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

mojavelinux
Administrator
In reply to this post by chloerei
Rei,

On Fri, Jan 2, 2015 at 1:34 AM, chloerei [via Asciidoctor :: Discussion] <[hidden email]> wrote:
I checked location 0x91cf in NotoSans CJK, it's not the right '量' glyph. And checked same location in DroidSansFallback, it's right glyph.

But when I open NotoSans CJK in KDE's font viewer, enter example text '变量'(means 'variable'), it shows the right glyph. I don't understand the mechanism.

Thanks for verifying. So my intuition was correct, FontForge is messing up the mapping somehow.

I can understand why it might be messing up, just not exactly why. As I understand it, there is an effort to reduce the size of CJK fonts by reusing glyphs or parts of glyphs. This is a feature of OpenType, but not TrueType. Programs like FontForge are retrofitting these features into TrueType, but obvious errors persist.
 

Than I checked mplus1p-regular-multilingual.ttf, I found a problem: it contains '量', but missing '变'(0x53d8), it's a common characters too. And I use it to generate PDF for my document, it missing a lot characters. I think M+ have not enough feedback for Chinese, maybe it's good for Japaness and Korea.

Thank you for evaluating. That's extremely useful information to know. Thanks to your insight, I can now talk about M+ more intelligently :)
 
Chinese has a lot of characters, common characters may have several thousand, when contains uncommon characters it comes to several ten thousand, and different regions have different variants (http://blog.typekit.com/2014/07/15/introducing-source-han-sans/). So CJK fonts will be large.

This is where native support for OpenType is essential. Unfortunately, it's still very new and not well supported or understood yet. But every journey begins with a single step and we are starting to grow the knowledge of fonts in the Asciidoctor community so we can start to make more informed decisions and plans.
 
NotoSans CJK is the most complete CJK font I know (different regions still need to install different font packages, because all CJK  characters up to 65535 limit. http://www.google.com/get/noto/cjk.html), DroidSansFallback is enough for common CJK use, M+ maybe not enough for Chinese common use.

Again, excellent information. Perhaps we can start with a theme for Asciidoctor PDF that uses DroidSans as the primary font and continue to research getting NotoSans to work for the future. Would you be willing to open an issue in asciidoctor-pdf to add a cjk theme that uses DroidSans by default?
 

Thanks for pay attention for CJK, I hope this content will be helpful.

Absolutely. This information you have provided is extremely helpful. I definitely want Asciidoctor to be a great toolchain for all languages. That starts with getting the characters on the screen and is further supported by informed decisions. Thanks for taking the time to provide this information and for participating!

Cheers!

-Dan

--
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

chloerei
Reply | Threaded
Open this post in threaded view
|

Re: How to generate fonts

mojavelinux
Administrator
Thanks Rei!

-Dan

On Wed, Jan 7, 2015 at 11:56 PM, chloerei [via Asciidoctor :: Discussion] <[hidden email]> wrote:
Open a issues https://github.com/asciidoctor/asciidoctor-pdf/issues/82


If you reply to this email, your message will be added to the discussion below:
http://discuss.asciidoctor.org/How-to-generate-fonts-tp2358p2651.html
To start a new topic under Asciidoctor :: Discussion, email [hidden email]
To unsubscribe from Asciidoctor :: Discussion, click here.
NAML



--