Interesting--gutcheck_u reported several mismatched double quotes, but all were explainable, e.g. poem fragments inside a quoted paragraph, or two quoted paragraphs with an illustration between. On the other hand, there were many mismatched left/right single quotes--evidently DP's ASCII single quote to Unicode left/right single quotes needs some work. I'll do some more testing of gutcheck_u while WWing submissions. It would be very handy if someone could upgrade Gutspell, which doesn't work properly on texts containing Unicode quotes (single and double), em-dashes, etc. Al -----Original Message----- From: gutvol-d [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of James Adcock Sent: Thursday, February 26, 2015 10:24 AM To: 'Project Gutenberg Volunteer Discussion' Subject: Re: [gutvol-d] Unicode UTF-8 Compatible Version of Gutcheck Off the top of my head, try 48325, which demonstrates the handedness issue. From: gutvol-d [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of Al Haines Sent: Wednesday, February 25, 2015 9:37 PM To: 'Project Gutenberg Volunteer Discussion'; 'James Adcock' Subject: Re: [gutvol-d] Unicode UTF-8 Compatible Version of Gutcheck I'd like to give it a try. Can you upload it somewhere? Or you can send me a copy as a zipped attachment. And if you can mention the etext numbers of several of the files you tested it against, I can cross-check them with Gutcheck and Bookloupe (http://www.juiblex.co.uk/pgdp/bookloupe/index.html). Al -----Original Message----- From: gutvol-d [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of James Adcock Sent: Wednesday, February 25, 2015 7:49 PM To: gutvol-d@lists.pglaf.org Subject: [gutvol-d] Unicode UTF-8 Compatible Version of Gutcheck I've created a Unicode UTF-8 Compatible Version of Gutcheck, calling it gutcheck_u -- if anyone wants to try it. This was primarily an exercise in finding and changing 8-bit char coding dependencies to 16-bit widechar coding dependencies, but it did require some additional coding. Currently it is a somewhat Windows-dependent implementation, so if you want to run it on another OS it would take a little bit of work. I find it more pleasant to use if one's development file format of choice is UTF-8, and/or if one are doing such things as left-handed / right-handed quotes. Testing it on PG released files I am in fact finding a fair amount of left / right handedness errors that are not currently being discovered. Let me know if you want to try it, same distribution terms as the original. Jim Adcock jimad@msn.com

Huh. Not obvious to me what he is doing in bookloupe. Jim. From: gutvol-d [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of Al Haines Sent: Wednesday, February 25, 2015 9:37 PM To: 'Project Gutenberg Volunteer Discussion'; 'James Adcock' Subject: Re: [gutvol-d] Unicode UTF-8 Compatible Version of Gutcheck I'd like to give it a try. Can you upload it somewhere? Or you can send me a copy as a zipped attachment. And if you can mention the etext numbers of several of the files you tested it against, I can cross-check them with Gutcheck and Bookloupe (http://www.juiblex.co.uk/pgdp/bookloupe/index.html). Al -----Original Message----- From: gutvol-d [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of James Adcock Sent: Wednesday, February 25, 2015 7:49 PM To: gutvol-d@lists.pglaf.org <mailto:gutvol-d@lists.pglaf.org> Subject: [gutvol-d] Unicode UTF-8 Compatible Version of Gutcheck I've created a Unicode UTF-8 Compatible Version of Gutcheck, calling it gutcheck_u -- if anyone wants to try it. This was primarily an exercise in finding and changing 8-bit char coding dependencies to 16-bit widechar coding dependencies, but it did require some additional coding. Currently it is a somewhat Windows-dependent implementation, so if you want to run it on another OS it would take a little bit of work. I find it more pleasant to use if one's development file format of choice is UTF-8, and/or if one are doing such things as left-handed / right-handed quotes. Testing it on PG released files I am in fact finding a fair amount of left / right handedness errors that are not currently being discovered. Let me know if you want to try it, same distribution terms as the original. Jim Adcock jimad@msn.com <mailto:jimad@msn.com>

Bookloupe supports UTF8, but does the same checks as Gutcheck. It doesn't check Unicode quotes. No idea what its developer's plans may be for adding such functions. There's a thread for it in DP's Tool Development forum (http://www.pgdp.net/phpBB2/viewforum.php?f=13). It's one of the Sticky topics. Al -----Original Message----- From: gutvol-d [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of James Adcock Sent: Thursday, February 26, 2015 10:30 AM To: 'Project Gutenberg Volunteer Discussion' Subject: Re: [gutvol-d] Unicode UTF-8 Compatible Version of Gutcheck Huh. Not obvious to me what he is doing in bookloupe. Jim. From: gutvol-d [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of Al Haines Sent: Wednesday, February 25, 2015 9:37 PM To: 'Project Gutenberg Volunteer Discussion'; 'James Adcock' Subject: Re: [gutvol-d] Unicode UTF-8 Compatible Version of Gutcheck I'd like to give it a try. Can you upload it somewhere? Or you can send me a copy as a zipped attachment. And if you can mention the etext numbers of several of the files you tested it against, I can cross-check them with Gutcheck and Bookloupe (http://www.juiblex.co.uk/pgdp/bookloupe/index.html). Al -----Original Message----- From: gutvol-d [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of James Adcock Sent: Wednesday, February 25, 2015 7:49 PM To: gutvol-d@lists.pglaf.org Subject: [gutvol-d] Unicode UTF-8 Compatible Version of Gutcheck I've created a Unicode UTF-8 Compatible Version of Gutcheck, calling it gutcheck_u -- if anyone wants to try it. This was primarily an exercise in finding and changing 8-bit char coding dependencies to 16-bit widechar coding dependencies, but it did require some additional coding. Currently it is a somewhat Windows-dependent implementation, so if you want to run it on another OS it would take a little bit of work. I find it more pleasant to use if one's development file format of choice is UTF-8, and/or if one are doing such things as left-handed / right-handed quotes. Testing it on PG released files I am in fact finding a fair amount of left / right handedness errors that are not currently being discovered. Let me know if you want to try it, same distribution terms as the original. Jim Adcock jimad@msn.com

On 2/26/2015 1:30 PM, James Adcock wrote:

Message

Huh. Not obvious to me what he is doing in bookloupe.

Bookloupe was intended to be the UTF-8-aware replacement for gutcheck. However, if I remember correctly, at the time the author wrote it PG wanted it to report all the same errors as gutcheck would or they would not agree to switch over to using it. So it ended up only partially UTF-8-aware, as with full UTF-8 awareness it wouldn't report some errors that gutcheck would have reported. But as far as I know Bookloupe is what PG now uses, not gutcheck. -- Walt

Actually, the WWers pretty much have had to use both since PG decided to accept only a single text file of whatever character set was needed to express the source material. To use 48325 as an example, Gutcheck sees this line (line 27 of the submitted file): _“THE WIDE, WIDE WORLD,” “QUEECHY,” like this: _“THE WIDE, WIDE WORLD,” “QUEECHY,” resulting in a "missing space" warning. If the line is too long, Gutcheck will also issue a "line too long" warning. Bookloupe sees the line properly, but doesn't/can't check the quotes. Gutspell sees the strange characters as part of the word, and flags the word as a misspell. This isn't limited to quotes--words adjacent to, or that contain, Unicode em-dashes, apostrophes, or ellipses cause the same problem. To check quotes, and to have Gutspell work properly, the WWers have to use Unitame to generate a temporary Latin1 text file from the UTF8 file, and Gutcheck/Jeebies/Gutspell the Latin1 file. As a WWer, I don't mind UTF8 text files that contain necessary Unicode characters, e.g. Greek, etc, but UTF8 text files that contain only Unicode quotes, apostrophes, em-dashes, and similar NON-necessary Unicode characters, are just a nuisance. If a text file doesn't need Unicode characters, submit a Latin1 text file. It's a lot easier on the WWers. Al -----Original Message----- From: gutvol-d [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of Walt Farrell Sent: Thursday, February 26, 2015 11:07 AM To: Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] Unicode UTF-8 Compatible Version of Gutcheck On 2/26/2015 1:30 PM, James Adcock wrote: Huh. Not obvious to me what he is doing in bookloupe. Bookloupe was intended to be the UTF-8-aware replacement for gutcheck. However, if I remember correctly, at the time the author wrote it PG wanted it to report all the same errors as gutcheck would or they would not agree to switch over to using it. So it ended up only partially UTF-8-aware, as with full UTF-8 awareness it wouldn't report some errors that gutcheck would have reported. But as far as I know Bookloupe is what PG now uses, not gutcheck. -- Walt

Depends on the character(s). I occasionally encounter Unicode characters in my projects, and their handling is a judgement call. For example, I render oe-ligature as either "oe" or "[oe]". Greek words or short phrases, I transliterate. (Lots of Greek, I avoid.) I recently did a book that contained symbols for the sun and planets, but the book didn't explain what the symbols meant--I had to research which symbol meant which planet. I decided to render the symbols as words, e.g. [Sun], [Earth], etc, figuring that would be easier on the average reader. PG's posting software, as opposed to its text-checking software, can handle any UTF8 file (text and HTML) in any language. I've posted all manner of Greek, Chinese, Japanese, and even a couple of Tagalog submissions. As for "upgrading the toolset", none of the WWers has the programming skills to upgrade software like Gutcheck/Jeebies/Gutspell to properly handle UTF8 files, check curly quotes, etc, etc. Al -----Original Message----- From: gutvol-d [mailto:gutvol-d-bounces@lists.pglaf.org] On Behalf Of James Adcock Sent: Thursday, February 26, 2015 5:21 PM To: 'Project Gutenberg Volunteer Discussion' Subject: Re: [gutvol-d] Unicode UTF-8 Compatible Version of Gutcheck

If a text file doesn't need Unicode characters, submit a Latin1 text file.

Whenever I've checked, I find I have a couple chars that fall outside of Latin1. Do I pimp the submission then, or god-forbid should the WW'ers upgrade their toolset?

As for "upgrading the toolset", none of the WWers has the programming skills to upgrade software like Gutcheck/Jeebies/Gutspell to properly handle UTF8 files, check curly quotes, etc, etc.

Granted, Windows-only right now, but I did gutcheck_u [Unicode version of gutcheck], and now I am sending Al a gutspell_u [ask me if anyone else wants it.] I don't think it's hard to get you better tools if you ask for them on this forum. Just don't say for example "it's got to work exactly like gutcheck except on Unicode" - because that description is an oxymoron. Firstly it's hard to track down and exactly duplicate all the bugs that are already in there, and secondly, the checks one would want on a Unicode file are not identical to the checks one would want on an ASCII file.