September 2005 - gutvol-d - lists.pglaf.org

ok then, let's go to work (part 2)
by Bowerbird＠aol.com 07 Sep '05

07 Sep '05

today's lesson will bear on distributed proofreaders. as a reminder, this thread is about the scanset created by jon noring for his "my antonia" demo: > http://www.openreader.org/myantonia since the scans themselves are a fairly hefty download, at 31 megs, you can grab the 5-meg djvu version instead. that allows you to follow along if i refer to certain pages. i've uploaded the text-file that finereader v7 generates as o.c.r. output for "my antonia". this text-file can be downloaded from this u.r.l.: > http://snowy.arsc.alaska.edu/bowerbird/myantonia1.txt or, for those of you who would prefer a .zip version instead: > http://snowy.arsc.alaska.edu/bowerbird/myantonia1-txt.zip i will probably make some references to this text-file in coming days, so if any of you wanna follow along, you should download it and become familiar with it... (and it's only a 500k download, under 200k for the .zip.) one of the most immediate ways that online scans can facilitate the goals of project gutenberg is to make it possible for people to check the text against the scans themselves, to make sure that it's accurate. generally, this is what distributed proofreaders does -- present the text alongside the scan so it can be proofed. but the relevance of the parallel is much more specific; for that we need to delve deeper into the o.c.r. output... as i understand it, d.p. has recently switched to a new methodology, which separates proofing and formatting into separate rounds (with 2 rounds for each of them). part of the formatting involves "meta-formatting". (i think this is part of a formatting round, anyway, not a distinct and separate "round" of its own, but if i am wrong about any of this, i assume that one of the people from d.p. will step in and correct my error.) one aspect of the "meta-formatting" is a checklist that, among other things, indicates if a chapter-heading exists on a specific page. this checklist is used to ensure that the proper markup gets applied to that chapter-heading. that is, each page-scan is pushed out to human being, who then marks an item on a list if it has a header on it. (and other items on the list if it has those other features.) as i have indicated in the past, it is not difficult to write computerized routines to sniff out these section-headers; so it is simply a ridiculous waste of valuable resources to have human beings be making this determination instead. it is much smarter to use the computer to do the bulk of the work at the outset, and then have a human check it. appended is the output from such a routine i wrote in just a short time -- the program is under 50 lines long. if you check against the "my antonia" text, or the scans, you will see that this routine has successfully identified the pages in the book on which there is a section-break. it gives you the page-number, and does the best it can to tell you the actual _text_ of the header on that page. programs like these only take a few minutes to run, so it's easy to see this is a more efficient way to proceed, compared to having a person drudge through every page. i won't tell you exactly _how_ this program operates, because it would do you good to look at pages where there is a section-break, and come up with an answer. a hint for you is that there are _numerous_ indicators, any one of which is sufficient in this current example... you might remember that i have a 30-item checklist consisting of dimensions that are indicative of headers. how many of the 30 items can _you_ come up with? feel free to share your answers with the whole listserve. if enough people come up with enough of the indicators, i will share the source-code of the routine i wrote here... in fact, just for fun, i wrote another quick little routine, using another one of my 30 indicators, and that routine gave me the output that i appended in the second p.s. as you see, this routine produced excellent results too. i have said it before, but i'll repeat it again here now: headers are specifically _designed_ to draw attention, so it is easy to locate them, even in raw o.c.r. output. but heck, before you know it, you'll be smart enough to figure out how to determine _other_ structures as well... another item on the "meta-formatting" checklist is _footnotes_. there is only one footnote in this text, but based on that, how would you write a routine to identify any footnotes? how about block quotations? expressions in a foreign language? tables? lists? poems? all of the various aspects contained in plays? it's not hard. give it a try... -bowerbird p.s. here's that output... 3 Book I 9 II 21 Ill 31 IV 36 V 42 VI 48 VII 57 VIII 70 IX 80 For several weeks after my sleigh-ride, we 91 XI 96 XII 101 XIII 108 XIV 119 XV 131 XVI 137 XVII 145 XVIII 156 XIX 160 Book II 162 Book II 168 II 176 Ill 181 IV 193 V 197 VI 206 VII 220 VIII 225 IX 233 It was at the Vannis* tent that Antonia was 238 XI 244 XII 258 XIII 264 XIV 280 XV 288 Book III 290 Book III 298 II 307 Ill 315 IV 332 Book IV 334 Book IV 342 II 346 Ill 361 IV 366 Book V 368 Book V 399 II 415 Ill 441 COPYRIGHT, 1918, BY WILLA SIBKRT CATHKR 443 CONTENTS 445 INTRODUCTION p.p.s. and here's the output from the second routine. i've left a big hint in here about how this routine operates; can you figure it out? 1=2 2=2 3=24 4=30 8=29 9=26 20=20 21=26 30=28 31=25 35=29 36=26 41=23 42=26 47=17 48=25 56=7 57=26 69=17 70=26 79=6 80=25 90=16 91=26 95=18 96=25 100=13 101=26 107=10 108=27 118=23 119=26 130=8 131=25 135=28 136=22 137=26 138=30 144=25 145=26 155=26 156=26 159=27 160=2 161=3 162=2 163=24 164=30 167=27 168=26 175=16 176=25 180=24 181=26 192=13 193=26 196=12 197=26 205=25 206=26 219=10 220=27 224=29 225=25 232=18 233=25 237=22 238=26 243=19 244=26 257=6 258=26 263=17 264=26 279=28 280=26 288=21 289=3 290=2 291=24 297=18 298=26 306=28 307=26 314=29 315=26 331=29 332=17 333=3 334=2 335=25 336=30 340=29 341=16 342=26 343=30 345=10 346=26 359=29 360=8 361=26 362=30 365=20 366=2 367=3 368=2 369=24 370=30 398=20 399=26 414=8 415=25 419=25 420=4

1 0

re: [gutvol-d] ok then, let's go to work (part 1)
by Bowerbird＠aol.com 07 Sep '05

07 Sep '05

marcello said: > Very wise. You know that you are not up for the challenges I put to you. ok, folks, here i will begin my practice of nonresponsiveness to such baiting and attempts to derail a productive thread into pettiness... -bowerbird

1 0

re: [gutvol-d] ok then, let's go to work (part 1)
by Bowerbird＠aol.com 07 Sep '05

07 Sep '05

marcello said: > what an original idea ... yeah, i thought so. strangely, jon noring didn't do that, though. he grabbed an existing version of the text, and proofread _that_ against the scans... that's why he's still got _errors_ in his text, errors i found by comparing that text with the output that i got from finereader's o.c.r. (thus, the .pdf does not contain jon's errors...) crosschecking independently-derived e-texts in this manner is a great way to do proofing. > Now, as you yourself said, the pdf format is > useless for further editing. well, actually, i didn't say that. because there are ways now to edit a .pdf. it's still clumsy, though. plus, if you were to correct the errors under the finereader interface, and _then_ generate the .pdf, the output would be fine. mostly i don't care much for the .pdf format. but some people seem to like it, even prefer it. truth be told, in this case, where each page of the book will fit comfortably on a monitor -- a two-page spread fills the screen nicely -- .pdf is a perfectly good way to read this book. and as the world moves toward tablet p.c.'s, some disadvantages of acrobat will fade away. (except for its inability to copy out clean text, which might be a problem that adobe finds is extremely difficult to overcome, given its m.o.) but the reason i commented on the fact that finereader does an excellent job of capturing the _formatting_ is that -- believe it or not -- too many digitization efforts, including d.p.!, often just _throw_away_ all that formatting, outputting o.c.r. results to raw-ascii plain-text. this is such a waste! there is extremely valuable information in that formatting, information about the _structure_ of the e-text. indeed, in a good many ways, the art of typesetting is the communication of information about the text's underlying structure. (in a phrase, "some content is _in_ the presentation".) > So why don't you try to output something useful, > like, say, an XML file. since you are one x.m.l. booster here, why don't you? and then take that x.m.l. output and run it through all your xslt conversions, and we'll see what we get! i'm curious to see how robust your methodology is... i highly suspect, though, that you will find that the x.m.l. that finereader outputs is a kind that you are unable to use. and _that_ box of problems is one you don't want to open. it's much more amenable to your x.m.l. hype machine to continue to pretend that any kind of "x.m.l." can be mapped to any other kind of "x.m.l." of course, if that was _really_ true, y'all wouldn't be having the .tei vs. .xhtml vs. .docbook discussions, you'd just use one format and convert to the other. > If you were able to get a TEI file out of your finereader, > you could actually and if pigs could fly, we could use them for air transportation, and not worry about muslim terrorists hijacking the "planes"... (i hope muslims will forgive me for that bit of pig humor...) ;+) > If you were able to get a TEI file out of your finereader, > you could actually proofread the thing and produce a pdf file > out of it that looks a *lot* better than the one you posted. well, as one of the .tei boosters here, why don't _you_ do that? then we'll compare your .pdf with my .pdf -- the one i create after having proofed the text and created a .zml file out of it -- and we'll see which .pdf looks the best of all... my .pdf will be up within the next two weeks, depending on how quickly i get through the thread i have just created, so you've got a little time to do your .tei markup. be my guest... > Besides you would get an html > and a plain text file into the bargain, ready for posting. finereader will create an .html file too, automatically. i'll be uploading that in a few days as well for you to see. and it does a pretty good job of creating an html-book, at least from the standpoint of how the thing _looks_. and of course, it will output a plain-text file. that one will be uploaded tomorrow. it'll also create an .rtf file, which is important because that retains much of the formatting information that i just noted is very important in ascertaining structure. so i'll upload that eventually as well. and i'll be making commentary on all of these uploads. (heck, finereader will even create a powerpoint-book. if only all these formats weren't plagued by scannos.) so fasten your seatbelts, folks, it's gonna be a ride... -bowerbird p.s. i probably won't bother responding to marcello much more, since most of his comments are usually not just not worthwhile, but an actual waste of time. so if he makes a point that you want me to address, it would probably be best if you'd reiterate that point, in your own words. in which case, i'll be happy to reply.

2 1

ok then, let's go to work (part 1)
by Bowerbird＠aol.com 07 Sep '05

07 Sep '05

ok then, let's go to work... greg asked for actual examples of how online scans might be used to good effect by project gutenberg. one good set of scans is that created by jon noring, for his "my antonia" demo. those scans are here: > http://www.openreader.org/myantonia i subjected the scans to o.c.r. using finereader v7. one of the options in finereader is to output a .pdf. i did that, and have uploaded that .pdf for your perusal: > http://snowy.arsc.alaska.edu/bowerbird/myantonia1.pdf if you look at this .pdf, you will find that finereader does a rather amazing job of retaining the book's formatting. of course, there are scannos in the text, which makes it unusable, but from a _formatting_ standpoint, it is fine. if the .pdf we create at the end of our digitization would look as good as this one finereader makes automatically, just from the scans, we could feel proud of ourselves... that's enough for today... tomorrow we'll go on to look at the o.c.r. output itself... -bowerbird

4 3

re: [gutvol-d] ok then, let's go to work (part 1)
by Bowerbird＠aol.com 06 Sep '05

06 Sep '05

josh said: > Personally, I think our TEI -> PDF output looks a lot better. i'll take a look. (i just explained why i admired the .pdf output of finereader. given that it takes absolutely _no_ work at all, being fully automatic, i think the job that it does is _excellent_. but no, that doesn't mean i admire it in an _absolute_ sense; in addition to the scannos, its font recognition can be wacky.) i'll assume you're open to feedback on these, and i'll give it, so let me know if you'd prefer not to have any... some of the .html-books that have come out of d.p. lately have looked really good. unfortunately, given the problems of viewing an e-book inside of a browser, there are too many situations where the beauty is hidden by browser-based flaws. (probably the biggest thing that irritates me is when a picture is split across a window, and i have to nudge it up or down to view it in its entirety. as a constant bugaboo, it's maddening.) but aside from the shortcomings of the browser, i'd say the people at d.p. are doing great work... -bowerbird

1 0

is that it?
by Bowerbird＠aol.com 02 Sep '05

02 Sep '05

is that it? snide comments from marcello and josh, in the latest thread, and that's all? maybe you people here are getting smarter about picking your battles... or perhaps katrina has given you a more-refined sense of priorities... at any rate, if you can be well-behaved, perhaps you deserve a bit of pudding... have a nice long weekend, everyone... oh, if anyone knows where george bush can find some of his military helicopters, to help those poor people in new orleans, give him a call, would you please? thanks. -bowerbird

1 0

re: [gutvol-d] ANNOUNCEMENT: XML has hit the PG archives! (gutvol-d Digest, Vol 13, Issue 19)
by Bowerbird＠aol.com 02 Sep '05

02 Sep '05

lee said: > Edit the .xml file with a simple text editor > (beware Microsoft tools!) to add the line: > ?xml-stylesheet href="persistent.css" type="text/css"? > You can experiment by adding new styles to 'persistent.css' > (don't forget to save the file and reload your browser after > adding rules). For example, add > "p { display:block; text-indent: 3em }" > and all of a sudden you will get distinct, indented paragraphs > (and some non-paragraphs will also become distinct and > indented). Add "teiHeader { display: none }" and all the > Gutenberg legal cruft, together with the metadata which is > typically only of interest to archivers, will disappear > (it's still there, it's just not "in your face" anymore). that is, in other words, if i tell it to use a stylesheet, and then go and create that stylesheet, it will work. :+) i knew that anyway, but i guess it's good to be reminded. ;+) *** jeroen said: > You can render XML, using XSLT + CSS in Firefox and IE, > for a small demo, look at > http://www.gutenberg.org/files/11335/11335-x/11335-x.xml. yes, i should have mentioned jeroen's files work in firefox... (not in safari. but in firefox.) > Some have argued (with valid reasons) that > the entire idea of TEI markup is broken, and > have proposed systems in which the mark-up is > separated from the text (stream of characters), > in such a way that multiple, parallel systems of > mark-up can exist. Think of a separate (part of a) file, > saying characters 21 to 34 are italics, and so on. > This may sound odd, but it is the way the old > Macintosh wordprocessor MacWrite worked. actually, that's the way the underlying _editfield_ of the (classic) mac operating system is structured. -bowerbird

5 6

Re: [gutvol-d] PG-TEI: feature request (gutvol-d Digest, Vol 14, Issue 2)
by Lee Passey 02 Sep '05

02 Sep '05

Robert Shimmin <shimmin(a)uiuc.edu> wrote: > Joshua Hutchinson wrote: > > > Give me a little better explanation of red-ink (and maybe a link to > > an example so that I can see what you mean too) and I'll try to see > > if we have anything currently that would work. > > Not much to explain. Historically, printers used red ink to > highlight important things. The most famous example being that in > many editions of the Gospels, words spoken by Christ are printed in > red. Mostly, I'm just asking for control over font color. > > I asked for red in particular because if there are philosophical or > technical reasons why the markup should not allow arbitrary control > of font color, it would at least be nice to be able to use red. > > -- RS I think you have answered your own question here. TEI is designed to not presume or dictate any particular presentation, with perhaps the exception of distinquishing between block and inline elements. If text is rendered in red text in the original, and you believe that indicates highlighting but you don't know why, use the <hi> element. If you want to document that the original highlighting was in red (without suggesting that it ought to ultimately be rendered in a red font) you can use <hi type="redtext">. If you want to suggest that the text ought to be rendered in a red font you can include a style sheet with the selector "hi[type~=redtext] { color: red }." If you want to be specific that you want the text to be rendered in red, without indicating that it was red in the original you could even use <hi rend="red">. TEI does not have any controlled vocabulary for the 'rend' attribute (although a number of commonly used values are emerging) so as long as your style sheet knows what "rend='red'" means, you ought to be OK. On the other hand, if you know why the text is being highlighted (i.e. it is the words of Jesus) you can use a more specific markup such as <sp who="Jesus">. Again, if you want Jesus' words rendered in red, use a style declaration such as "sp[who=Jesus] { color:red }". Despite TEI's acceptance of the 'rend' attribute, if you want to suggest how the document should be rendered you really should use fairly specific and unambiguous markup combined with Cascading Style Sheets or XSL transformations.

1 0

Re: [gutvol-d] PG-TEI: feature request.
by Joshua Hutchinson 02 Sep '05

02 Sep '05

There is a rend="font-family(x)" that will work for at least some of that. I know it would work for the roman, I think it would work for blackletter (that is a font change right?). I'm not sure what red-ink means, so I don't know if that one would be covered. (NOTE: This rend is mentioned in section 6.1 in the guideline file) Give me a little better explanation of red-ink (and maybe a link to an example so that I can see what you mean too) and I'll try to see if we have anything currently that would work. Josh ----- Original Message ----- From: "Robert Shimmin" <shimmin(a)uiuc.edu> > > I'm looking at Version 0.3 of PG-TEI. If that is no longer the current > version, apologies. The following are highlighting modes (inline 'rend' > values) that I've seen employed with some frequency in public domain printed > materials, that are not presently supported by PG-TEI. > > roman (when the 'default' typeface is not roman) > blackletter > red-ink > > Thanks, > -- RS > _______________________________________________ > gutvol-d mailing list > gutvol-d(a)lists.pglaf.org > http://lists.pglaf.org/listinfo.cgi/gutvol-d

2 1

re: [gutvol-d] banana-cream
by Bowerbird＠aol.com 02 Sep '05

02 Sep '05

marcello said: > Your reader is a very inferior solution to XML, > which can generate formats you can > read on all known platforms. then why don't you get to work on your precious x.m.l.? -bowerbird

1 0