Steve Thomas <stephen.thomas@adelaide.edu.au> writes:

But create an HTML file, properly, and then the user can do what they like with it. Want to print it out in Georgia 24pt? No problem. Your choice.

HTML isn't the best choice if you are interested in printing. Define "properly created" :) XML plus a customizable stylesheet (XSL or DSSSL) is better. For those who do not want to create a printable PDF file on their own offer a pre-generated PDF file. -- | ,__o | _-\_<, http://www.gnu.franken.de/ke/ | (*)/'(*)

IMO: HTML isn't the best choice if you are interested in anything.

This obsolete format should not be considered if even mentioned. Use XHTML + CSS instead, if you are allergic to XML.

XHTML is HTML 4.0 designed to work as an XML application. In fact, there aren't a lot of differences between HTML and XHTML. Of course, XHTML, HTML, and XML are all "children" of SGML anyway, so we're all talking about generalized markup in some form or another. You can't set up a rigid set of rules that will apply across all past, present and future versions of printed works in electronic format. Whatever format you choose to use, must be extensible enough to scale for future capabilities, as well as the ability to handle the capabilities of documents created in the past.

With properly written XHTML and customizable CSS user can do what ever he wishes with the files and the things are still as they should be.

Almost whatever s/he wishes. There are limitations in every format, depending on how broadly you want to consider using it. David A. Desrosiers desrod@gnu-designs.com http://gnu-designs.com

As usual, people have missed the point of the original post (Anne's) which was that we need to remember the *user* -- that guy in Africa with only 2 hours of electricity each day. Anne suggested (I think) that he uses a laptop, but more likely he's using a worn-out IBM 486 running Windows 3, so all this geek-talk about XML and XSLT etc. is irrelevant to him -- he'll be lucky if he can run a standard web browser. [I can't believe that people still think they're doing good by shipping old 486's to Africa -- but apparently its true. I recently donated some old Pentium II's to a charity, and they couldn't believe their luck.] Anyway: Karl Eichwalder wrote:

Steve Thomas <stephen.thomas@adelaide.edu.au> writes:

...

But create an HTML file, properly, and then the user can do what they like with it. Want to print it out in Georgia 24pt? No problem. Your choice.

HTML isn't the best choice if you are interested in printing. Define "properly created" :) XML plus a customizable stylesheet (XSL or DSSSL) is better. For those who do not want to create a printable PDF file on their own offer a pre-generated PDF file.

My definition of "properly created" HTML would be HTML4 strict, plus CSS. I was trying to avoid obvious detail. And HTML is the *best* choice for printing if you don't have the in-depth knowledge about XML/XSL etc. or the tools to make that happen. Anyone with IE6 can make a pretty good print of my HTML books, straight from the browser. Skippi wrote:

This obsolete format should not be considered if even mentioned. Use XHTML + CSS instead, if you are allergic to XML. With properly written XHTML and customizable CSS user can do what ever he wishes with the files and the things are still as they should be.

XHTML is -- for practical purposes -- the same as HTML 4 strict, except that it enforces good practice, whereas HTML allows the author some latitude. The important difference, to a user, is that HTML is pretty much guaranteed to work in all browsers, whereas XHTML can be "difficult" in some circumstances -- e.g. if you include the <!xml> header, it can fould up IE6. A while ago, I started converting all my ebooks to XHTML, but immediately ran into problems that were'nt worth my time to fix. One day, browsers will commonly deal correctly with XML of any type, with an appropriate style sheet. Right now, HTML is the format that works best. This is, I know, very boring to those of us who like playing with the latest gizmos and formats. But the reality is, if you want the widest possible audience, you've got to give them the format that's easiest for them. For a more detailed discussion of this topic, see the archives, about January this year if memory serves, where you'll probably find me -- and you -- saying much the same things. -- Stephen Thomas, Senior Systems Analyst, Adelaide University Library ADELAIDE UNIVERSITY SA 5005 AUSTRALIA Tel: +61 8 8303 5190 Fax: +61 8 8303 4369 Email: stephen.thomas@adelaide.edu.au URL: http://staff.library.adelaide.edu.au/~sthomas/

Karen Lofstrom <lofstrom@lava.net> writes:

On Wed, 20 Oct 2004, Steve Thomas wrote:

...

[I can't believe that people still think they're doing good by shipping old 486's to Africa -- but apparently its true. I recently donated some old Pentium II's to a charity, and they couldn't believe their luck.]

My Linux users group installs thin client computer labs for schools. We happily accept PIIs, but turn down 486s. We use PIIs and PIIIs as thin clients, removing the hard drives and installing bootable NIC cards, and connect them to a fast server running K12LTSP Linux. We can create a usable 30 client computer lab for $3000 or so, since the clients are all donations.

I can't speak for Africa, but I have spent the last 14 years living in the deepest parts of China, Laos, and Cambodia. In the last 7 years I have not seen anything older than a PII except in a few old government systems and ancient bank computer networks running OS/2. And I'm not talking about the big cities like Vientiene, I'm talking about villages which barely have electricity and other odd corners with flakey old generators grumbling in blackend soot covered back sheds. Just because the electricity is only on for a few hours a day doesn't mean that people don't have access to okay technology. Hell, I've seen rice farmers along the Mekong River using picture phones to send pictures of babies to relatives in Bangkok. The third world ain't always as backward as people in the first world think. There are large areas that are that bad, but then ebooks will not be an option for them until they have bridges connecting them to settled areas, or proper water, bottled gas for cooking.... b/ -- Brad Collins <brad@chenla.org>, Bangkok, Thailand

Ack! This is a looong post.... and I'd promised myself I wouldn't get dragged into this flame-fest :( Steve Thomas <stephen.thomas@adelaide.edu.au> writes:

OK, you've somewhat overstated the case, and I think by now we'd all agree that "8-bit" characters are important. But it is a shame that most of the geeks -- no offence, I count myself as one -- on this list, immediately skipped your main point to whine about the need for accents and foreign scripts. You guys can't seem to see the wood for the trees.

You're right, it's not just about accents, and it's not just about consistently converting texts into different formats, though these are both important issues in their own right. This aside, it's you who have it backwards. You keep talking about the end-use of the text, which is opening up a file and reading it. But it's far from being this simple. XML is not meant for humans, it is meant for software. The XML will be converted to plain-text, HTML and PDF for humans but mostly the XML will be used by applications humans need to find texts and determine if they are worth reading in the first place. If you have a small library with 10,000 books in it, and the library is shelved roughly by category you can easily get to know it just by glancing over the spines. You could even have a rough list that breaks down the books by title, author and category. But if you have 100,000 or a 1,000,000 books in your library your job of finding things becomes a lot more difficult. Keyword searching ala Google fill never cut it. Google gives you a means of finding your car keys -- you know what you are looking for and you ask it to look for places which it thinks might have them. Ask Google for a list of the works by Charles Dickens and you will get a list of web pages it thinks has lists of Charles Dickens' works. Ask the LOC Catalog the same question and it will return you a list of items in their catalog which claim to have been written by Charles Dickens. But this list would be huge because of duplicate editions of individual works. A Christmas Carol alone turns up a couple hundred items. But what if you could ask this same question and it would return a list of works (not web pages, or different editions) by Charles Dickens organized in any way you want? But this is not a good example. Can you ask for a list of all the characters in Great Expectations? Can you search for all contemporary obituaries of Charles Dickens? To build applications which answer these types of questions requires more than a good cataloging system (though the FRBR approach goes a long ways in this regard) you need the table of contents of each work (a TOC is description of the structure of a text) and you need to have a good index of what is in the text. A back of book index is more than just a matter of keywords, it is a form of semantic markup. It maps concepts, people, places and events to the text itself. By combining the catalog metadata, the table of contents, and a good quality index we have the basic tools for finding a book and determining if it is worth reading. We do this today in libraries but it is a slow laborious task which requires you going to a catalog looking for possible candidates, then retrieving each candidate and scanning it's TOC, preface, dust-jacket blurb or introduction or index to determine if it's worth reading. Traditional libraries are restricted by the physical medium that books are published in. But if you could pull all of these elements together into a consistent framework, you would have a remarkable resource which would transform an archive of books into a repository of knowledge which is far more valuable and powerful than the sum of its parts. Semantic markup like TEI is needed not only for creating this kind of library, but for creating services which will be needed as the amount of information on the Net grows beyond what even monster search services like Google can handle. You talk about missing the forest for the trees but you forget that a large part of the forest is a tangled root system deep underground which the end user will never see. Without that root system the forest will die. Structured, semantic markup and rich cataloging are the root system of a library. Anyone who says -- I don't care about the technical stuff just give me what I want, doesn't understand that it's the technical stuff which enables them to get the stuff they want. Is this hard work? Hell yes, and it should be. Understanding, evaluating and making sense of the world around us is the most difficult thing humans do. But saying that it's not worth doing because it's hard is simply pathetic. Look at works like the OED. Would they have been created if their attitude was, oh, it's too hard to build a dictionary based on historical principles and I don't read the quotes much anyway, so just give me a list of words. Even if you don't read the quotes, the unabridged OED and the unabridged Websters, or Century Dictionary were used to create brilliant concise works like Merriam-Websters Collegiate, or the Concise Oxford English Dictionary. The OED and the massive collection of research and material that was created to write is the root system for all dictionaries Oxford produces. The more important question we should be asking is, what is the role that PG and even DP should be playing in all of this. It's reasonable to ask that PG produce basic structured markup which shows the basic structure and important elements in each text. This is no more difficult than HTML. I believe that a new group needs to be established who will then take the simple TEI produced by PG and DP and then doing more complex cataloging, indexing and semantic markup which will then be sent back to PG to be released as new editions. The TEI documentation (which is 1,400 pages -- not 14,000 as Bowerbird exaggerated) recommends that markup be done in several passes. Start with simple structural markup (as I said, is about the same as HTML), and then pass it onto another team which can do a second more detailed pass, and so on until its complete. In this way you have a means of creating texts which will be gradually woven into the library but everyone will be using a consistent and interoperable format which can be as simple or as complex as anyone requires. If everything is in basic TEI-Lite, it will be easy for smaller specialized groups to come along to do this additional markup. A group could form around a single author like Mark Twain, or around a category of works like mathematics. Then it will be easy for them to donate back their work to PG, making the texts richer, rather than their work becoming a separate branch of the texts which aren't interoperable with the PG editions. Plain-text, HTML and PDF can't do this because they are display formats for human consumption. Each have their uses and their markets. TEI is used for the root system which needs to be grown, tended and cared for as the forest grows, even if 90% of people aren't even aware it's there or don't understand that the applications they depend on to find any particular tree in the forest and see if it's the tree they need, wouldn't work without it. To understand where I'm coming from on all of this, I should mention (plug plug plug) that I've been working on just such a system (http://www.chenla.org) which is divided into two parts -- the Burr Metadata Framework (BMF) which is meant to be sort of a Wiki markup for both integration of and export to TEI and MARC. The second part is the Librarium which uses BMF to integrate the catalog with the works in a library. We have recently put up our first experimental record (an authority record Charles Dickens) which has been converted into html, and plain text. Conversion to TEI and MARC is coming. Taken together, the system can be used to integrate library catalogs with books and other texts and reference works all together with authority data for persons and groups, geographic locations, events and concepts. We don't intend to be a service for the general public but rather create a catalog and content for other use in other libraries and web sites. The site is hosted at ibiblio. In the next few weeks we should have enough documentation and another 30 or more records (which we call Burrs) online to make a general announcement of the project. I'm still ironing out some bugs in the version control software and still need to do a lot of work to complete a general introduction to the design but it's all getting there. At the moment we can convert BMF to Emacs-Wiki format which I then use to publish to Blosxom which delivers basic HTML. BMF was designed with conversion to TEI in mind, though this might seem hard to believe when you look at the BMF source the first time (there is a link to a pretty-print version of the Dickens source). So what's in it for PG? The Librarium will be developing detailed authority and bibliographic records for all PG material and it's hoped that PG can eventually draw on our catalog material for it's own authority records and catalog. This should be a help both for books already in PG's collection but also for copyright clearance for new books and free up resources for putting out more books, with better metadata. b/ -- Brad Collins <brad@chenla.org>, Bangkok, Thailand

--- Marcello Perathoner <marcello@perathoner.de> wrote:

Steve Thomas wrote:

...

Basically, Anne is right -- who cares about this stuff?

That is the exact same answer Tim Berners-Lee got when he first presented his stuff. :-)

Indeed. Over at DP we're progressing, in small baby-steps, toward producing decently marked up editions of all new material we produce. And when we at DP find a markup format we're comfortable with, then PG had better get comfortable with it as well, because we are now produce the vast majority of all PG material. -- Jon Ingram _______________________________ Do you Yahoo!? Declare Yourself - Register online to vote today! http://vote.yahoo.com

--- Marcello Perathoner <marcello@perathoner.de> wrote:

A simple XSLT will convert your format into TEI.

I'm not sure any use of XSLT can be called simple :). I've tried reading the spec, and I'm still recovering from the headaches. Fortunately there are easier ways to style (rather than transform) XML, using CSS. This is very well supported in all the Mozilla derivatives. While XSLT is something I'm going to have to look at eventually, for the moment I'm happy with CSS :). If you want an example of what I'm playing with at the moment: recently I and another DP volunteer have been kicking around some ideas for semantic markup of drama. While initially we were working with straight HTML, this quickly gets annoying, due to the amount of messing around with divs involved, and the need to consider how the output will be displayed on browsers with poor support for CSS. I've found it much easier to investigate options by working with an 'HTML+extra tags' markup. You can see my current working by looking at the blah.* files here: http://www.pgdp.net/phpBB2/viewtopic.php?p=94734 Save each file to the name given in its post subject heading. Any Mozilla derivative should show the .xml file styled in a way which almost exactly replicates the .html file. The source for the XML edition is much easier to read. Those of you who know TEI can probably tell that 'my' markup is very similar to TEI markup (although a little more verbose). Much of it was arrived at independently, which makes me more confident that this styling approach is relatively sensible. The example demonstrates markup of drama and poetry, with decent handling of line continuations and line numbers in poetry, and stage directions in drama. I've used the HTML 'edition' of this poetry markup for quite a while now in texts I've PPed for PG. Note that this is still a work in progress, so resist the tempation to criticise the minutiae of my CSS :). One of the other reasons I think a simple XML-style is useful is that we're currently planning to seperate the proofreading rounds from the markup rounds at DP. Every page of a DP project currently goes through two 'rounds' of processing. In each round proofers are expected to not only detect OCR errors, but add inline markup for italic, bold, material in non-Latin alphabets, etc., and add block markup for poetry, tables, and so on. This will be split into an initial two rounds only concerned with the text, plus an extra procedure to mark the text correctly. At the moment the markup we use is homegrown and kludgy -- we have a great opportunity at the moment to move to something more sensible, and I strongly believe that some simple XML-derivative is the markup we need. I'm even more convinced of the utility of XML for DP now that I've seen how easy it is to style it. One of the problems of relying on something like XSLT is that it can be hard to go backwards from errors in the output to find the corresponding error in the original XML input. Being able to get direct feedback by viewing a styled version of the XML makes life much easier. -- Jon Ingram _______________________________ Do you Yahoo!? Declare Yourself - Register online to vote today! http://vote.yahoo.com

Marcello wrote:

Jonathan Ingram wrote:

...

I'm not sure any use of XSLT can be called simple :). I've tried reading the spec, and I'm still recovering from the headaches. Fortunately there are easier ways to style (rather than transform) XML, using CSS. This is very well supported in all the Mozilla derivatives. While XSLT is something I'm going to have to look at eventually, for the moment I'm happy with CSS :).

CSS, while simpler, is less powerful and gives you only HTML.

CSS can be applied to any XML markup for viewing on web standards browsers, but because current CSS is limited there are certain HTML functions (tags) it simply won't be able to enable, or to enable cleanly. In CSS 'display', for instance, there is no value for identifying when some XML element represents an object/image, nor a hypertext link/anchor. Obviously, there is no 'display' value for an inline note because HTML never supported this. (We find markup for inline notes in the TEI and DocBook vocabularies. Note that CSS can move a span of inline text to the side in its own box, I've tested it out myself, but IE6 unfortunately does not recognize the needed CSS so in IE6 the inline note stays inline, not a good thing.) Of course, one can use XLink for object/image embedding and anchors (and XLink makes more sense anyway than using CSS since it is a vocabulary-independent means to embed objects and enable links), but then current web browsers are very deficient in XLink support (Mozilla has very limited XLink support -- haven't tested FireFox yet -- while IE and Opera have zero XLink support.) The OpenReader System 1.0, should it become a reality (and we are working on it -- we've made great strides in the last few weeks in garnering fairly high-level support), intends to fully support the more important parts of the XLink specification in version 1.0. We may also add one or more custom CSS values to 'display' to emulate links/ anchors, objects/images and inline notes (OpenReader will include a facility to open 'booklets' to display non-inline content, in part to support OEBPS which enables this cool ebook feature.) We also plan to investigate a future version of OpenReader to *natively* support TEI-Lite or some subset of TEI (including handling inline notes which will be trivial for OpenReader to handle.) We may even develop a next-generation styling language to address the deficiencies of current CSS2 and CSS3 but which doesn't have the complexity of XSLT/XSL-FO. The problem with CSS is its ties to the HTML paradigm and legacy support. In OpenReader, we are freeing ourselves from these legacy issues and thus can think outside the box and move on to the next generation web browser -- in essence to go beyond HTML. Jon Noring OpenReader: http://www.openreader.org/

--- Marcello Perathoner <marcello@perathoner.de> wrote:

I don't think we'll get PG to post texts in non-standard cooked-up formats.

Neither do I, and I don't want them to. Hopefully we'll either use TEI, or a markup which can easily and losslessly transformed into TEI. However, there are a lot of people out there, including a lot of DP volunteers, who are unconvinced about the utility of XML, and one of the best ways to *fail* to change their mind is to plonk 1400 pages of documentation in front of them and say 'here's what you should be using, come back when you've finished reading' -- this is true even of TEI-lite, which has some foibles you have to see past (overly terse tags, for example -- at least to my mind :) ). I used to be one of the members of the 'undecided about XML' camp myself. I've gradually changed my mind, and I'm working on helping to change the minds of those I'm working with. I also don't just automatically accept the TEI-way as being best for the applications I wish to use it for -- so I've been developing my own structured markup which I'm happy with, and which happens to have converged very closely to the corresponding TEI markup. As I said in my previous email, this makes me much more confident to accept the use of TEI-style markup in areas where I haven't had the time to investigate alternatives. Those of you who aren't involved in DP will probably see nothing more about this until the day that 85% of new PG ebooks come with a TEI edition :). -- Jon Ingram __________________________________ Do you Yahoo!? Yahoo! Mail - Helps protect you from nasty viruses. http://promotions.yahoo.com/new_mail

--- Marcello Perathoner <marcello@perathoner.de> wrote:

Jonathan Ingram wrote:

...

and one of the best ways to *fail* to change their mind is to plonk 1400 pages of documentation in front of them and say 'here's what you should be using,

Then don't do that.

You don't plonk the IBM PC Technical Reference Manual (5000 pages) in front of your secretary if you want her to type a few pages in M$-Word. You just give her a "Word for Dummies" book and that is all she needs. She don't need to know about the difference between AGP and PCI-X bus.

You're quite right. I let the current confrontational 'vibe' of this mailing list get the better of me. Sorry. The point I was trying to make is that there are many people, myself included, who need to be given real arguments in favour of using something like TEI, and who won't accept that TEI does things the right way just because it's been around for a while :). There's quite a few people like me at DP, and I imagine there are quite a few more reading gutvol-d. As I've convinced myself that, at least in the areas I've investigated, TEI's methods seem quite sensible, I'm more open to 'trusting' the rest of it... and I thought some people would be interested in joining me on this journey. As gutvol-d is being a little too confrontational for me at the moment, I'll probably go back to exhibiting my enthusiasm in the more congenial atmosphere of DP. -- Jon Ingram _______________________________ Do you Yahoo!? Declare Yourself - Register online to vote today! http://vote.yahoo.com

Jim Tinsley wrote:

Nobody has an objection to valid TEI texts, but valid TEI texts alone _are not enough_. An XML file that cannot be read (by an actual human) is as useful as a lock with no key.

Not so. Having a TEI text posted would enable third-party developers to come up with their own converter solutions eve if we didn't get very far with ours. There are a lot of people around who already convert the text files into other formats. Their jobs would get much easier.

I really no longer give any headroom at all to the approach "Post XML Now Because That Is The One True Way And We'll Figure Out How To Read It Later." If for no other reason, then because the most important part of the WW job is to check the texts before posting, and if we can't read it, we can't find the errors, and if we can't find the errors, we can't fix 'em.

A TEI text is basically a text file. So you can read it in any editor. If you use emacs you can also validate the TEI file against the DTD without leaving the editor. A perfectly valid TEI file with no spelling errors should be good enough to post. What you expect from us TEI developers is that we produce the 150% perfect solution before you even consider starting to post files. That is not the way software development works. And this attitude is in my opinion the main cause why we have gotten nowhere with TEI in the last 3 years. Lets start now with a version 0.0.1 of the TEI process. Of course at some later time we'll have to do all the posted files over again. Probably more than once. But its better than sitting here and playing with bowerbird because we are bored.

Next, we need a process, using open-source, cross-platform tools -- the standarder the better -- to convert that XML into, at a minimum, plain text and HTML. Other formats are welcome but optional. That process must work for _all_ teixlite files, not just ones that are specially cooked, using constraints not specified within the chosen DTD. Here's where we hit the rocks today.

TEI defines a standard way to extend the DTD. I used this standard way to extend the TEI DTD into what I called PGTEI. This still is a perfectly valid TEI DTD according to the TEI specs.

I don't want to imply specific means from which this process is to be constructed. Obviously XSLT is one possible approach, but I certainly do not want to imply limitations on what that process should use. The only things we must have -- both for our own internal practical purposes and for the use of future readers -- is that it should work reliably on _all_ texts that conform to the XML DTD chosen, be open source, and be cross-platform. A reader needs to be able to tweak the transform and re-run on her own desktop.

You misunderstand what a DTD is. It just gives you syntactical correctness. I can cook up a perfectly valid XHTML file which is semantically bogus: <div><h6>1</h6> <div><h5>1.1</h5> <div><h4>1.1.1</h4> ... </div> </div> </div> This is valid HTML (didn't bother to check) but will render not so well. You cannot build a conversion tool that will produce good results on all syntactically valid TEI files, like you cannot build a browser that will make sense out of semantically bogus HTML files. Furthermore TEI is geared towards marking up existent texts, so scholars can study the text without having to get the physical book. It is not so good as a master format for print processing. That's why I had to add some more tags and attributes to my DTD. (Which doesn't make any text that uses my DTD less standard, because TEI is expressly designed to be extensible. But I'm repeating myself.)

And just re-reading that last, when I say "must work reliably on ALL texts" I do not mean to imply that the same XSLT must be used for all texts, though obviously that would be of benefit, if we can manage it.

So why not start posting texts marked up in PGTEI, which will by definition work well in my conversion chain? And at the same time start posting Jeroens texts, which will convert fine in his chain? This way we could both start putting up an automatic online conversion chain. (The guy who did this already in Java has somehow vanished, so I think we have to start over again.) For the start I will act as interim Post-Processor for people wanting to post PGTEI and pass on to you only the perfectly good ones. You'll just have to stick in the etext number where I put 5 asterisks. I claim the .pgtei file extension, Jeroen can claim what extension he sees fit for his files. So we can have bith an alice30.pgtei and an alice30.jtei. -- Marcello Perathoner webmaster@gutenberg.org

Jon Noring wrote:

I probably missed it from one of your prior messages, but do you have your PGTEI documented anywhere? Have you put together an actual Schema/DTD which can be used to validate documents for validity to PGTEI? And a list of your custom vocabulary extensions?

Start here: http://www.gutenberg.org/tei/

Also, another question to ask is if it is documented anywhere how Jeroen's version of TEI compares with your PGTEI?

No. -- Marcello Perathoner webmaster@gutenberg.org

--- Jim Tinsley <jtinsley@pobox.com> wrote:

Correct spelling is necessary but not sufficient. I don't know about other people, but I most commonly find errors by skimming the text. I can't do that with XML.

As my post earlier on today indicates, this isn't true. Assume that PG starts accepting some TEI-related schema. All you need is a relatively simple CSS stylesheet, and you can open the XML and view it perfectly directly. See http://faculty.washington.edu/dillon/xml/ for some examples where you can view styled (XML-conformant) TEI directly in your browser, with no intermediate transformations required. -- Jon Ingram _______________________________ Do you Yahoo!? Declare Yourself - Register online to vote today! http://vote.yahoo.com

I was just reading over my last posting, hoping I wasn't the one sending bad vibes to Jonathan, who is exactly the kind of person we _need_ in a discussion like this, when I came across something else that Marcello said, that I didn't comment on first time round:

Lets start now with a version 0.0.1 of the TEI process. Of course at some later time we'll have to do all the posted files over again.

Now, please don't take this as a policy statement or anything, but I really, really HATE doing anything KNOWING that it's wrong and will have to be done again. I mean, bone-deep HATE it. Factor that in however you will. An argument against setting up an experiment in a production environment, or a personal foible? I report -- you decide! :-) jim

Jim Tinsley wrote:

...

Lets start now with a version 0.0.1 of the TEI process. Of course at some later time we'll have to do all the posted files over again.

Now, please don't take this as a policy statement or anything, but I really, really HATE doing anything KNOWING that it's wrong and will have to be done again. I mean, bone-deep HATE it.

*Why* is it wrong? If, having to redo something is an indication of premature start, we shouldn't have posted the first 10.000 books, because we have to repost them all. I have been architecturing and programming for 20 years now and I cannot remember one single instance I got it 100% right the first time. -- Marcello Perathoner webmaster@gutenberg.org

--- Joshua Hutchinson <joshua@hutchinson.net> wrote:

I'll be going over this with Jon when I can, but my early idea is that we work on a couple of DP e-texts (the two of us have TONS to choose from!) and improve the XML markup standard enough for basic work. In a few weeks or so, I'd like to get a few projects posted to PG that use XML (TEI) with a CSS style sheet in place of the normal HTML that we always produce on our projects. The normal text file will of course be created. Once we have a canon of TEI to work with, hopefully the developers out there can start working on tools to help produce HTML or TEXT or PDF directly from the master.

Just to put people's minds at rest, I don't believe we should post XML+CSS without (at the very least) an HTML edition -- certainly not until we have agreement on a common base of XML to use, and well tested tools to convert from this to (at the minimum) an HTML edition that displays acceptably on a wide range of browses. Even if we do end-up using a 'nonstandard' XML markup at DP, I agree with Joshua that we should try as hard as possible to ensure it can be converted easily to TEI (derivatives of which seem to be in favour around here). People at PG will not see the DP-internal markup, only our output, which will conform to the standards we will hopefully agree on at some point :). -- Jon Ingram __________________________________ Do you Yahoo!? Read only the mail you want - Yahoo! Mail SpamGuard. http://promotions.yahoo.com/new_mail

Jim Tinsley wrote:

I know of no CSS for Marcello's PGTEI. Perhaps one could be crafted for it.

It already works pretty well with Jeroens XSL: http://www.gutenberg.org/tei/examples/css/lmiss.xml I had to replace all named entities (like —) with numeric ones. I did that manually, so maybe I got some of them wrong. All quotation signs are missing because I replace quotation signs with <q> </q> and Jeroen does not. But this should be very easy to add to Jeroens XSL. -- Marcello Perathoner webmaster@gutenberg.org

Jim Tinsley wrote:

I really do not mean to be disrepectful when I -- speaking for myself -- say that I'm not interested in spending my time making developers' jobs easier. That's not what I'm here for.

One one hand you wonder why we developers are not able to come up with a solution on the other hand you are not disposed to get one inch back on your developer-unfriendly position. Your policy of not posting TEI files is at present the main roadblock. It's like requesting the final release of the product before allowing a beta test. I have been doing other (hopefully useful) work and have not looked at the TEI code for about a year now because I don't see a way to get it to work with this `moratorium' in place.

We have text, and HTML, both proven and well-supported formats that we know how to work with and for which we know there is a demand. I'll stick to those until we can see a way clear through to making successful XML.

You sure know how to work with PLAIN ALL CAPS ASCII TEXT FILES but that's not a reason to shun all progress since.

Correct spelling is necessary but not sufficient. I don't know about other people, but I most commonly find errors by skimming the text. I can't do that with XML.

After a few weeks you'll skim thru TEI like you skim thru plain text. (Use an editor that highlights the tags and use a low contrast color for the tags.)

And it may not be the way software development works, but then we're not a software development project.

But you depend on software. DP is 250.000 lines of code. If it was not for software you wouldn't have much to do.

that's not the problem. If the process we agree for teixlite is, say, run it through Saxon, then I expect to be able to run all teixlite files through Saxon, and not have a submitter say "oh, no, you must use Xalan for this file, and not just any Xalan, but one with my patch in it."

You have to use PGTEI stylesheets to convert PGTEI text. You can use them with any XSLT 1.0 compliant processor.

You see, we appear to differ very fundamentally on one point. It's my lock and key analogy again. I do not want to start down the road of producing posted files from an XML if the transform, will be, for any reason, not repeatable in a year's time, or five, or ten.

This amounts to the same as: never start at all. Remember: the first files were uppercase ascii. We *had* to do them over again. We *are* doing all pre-10K texts over again. We *will* have to do the TEI files over again, maybe more than once. That's only being realistic.

I do not want to start down the road of producing posted files from XML if an end-user who wants to -- on whatever platform -- cannot replicate the process.

Then you should also post all the scanned pages so a user can redo the OCR on her platform if she wants to. I think we can postpone this, because the user can grab the converted files. And if converting at home is an issue with him, hey!, the tools are Free Software. He can change them until they work on his platform and and submit the patches to us.

...

For the start I will act as interim Post-Processor for people wanting to post PGTEI and pass on to you only the perfectly good ones. You'll just have to stick in the etext number where I put 5 asterisks.

No; I, at least, don't want to work with an experimental process in which each text is an exception.

Is there some qualifying exam to become a whitewasher? I ask, because by now I'm so desperate that I'm quite willing to become a whitewasher myself just to see some TEI texts posted.

Why can't we just name them .xml? I see no reason to invent extensions. _Is_ there one? Not that it matters much, just curious why you would think this a good idea.

Because there ain't such a thing as an XML file. XML is just a framework for building applications. XHTML is an XML application, SVG is an XML application, TEI is an XML application, OpenOffice file format is an XML application ... Labelling a file .xml is like labelling a Word file .bytes -- Marcello Perathoner webmaster@gutenberg.org

Greg Newby <gbnewby@pglaf.org> writes:

Your claim that we need to start posting more stuff in XML in order to achieve the ** goal above does not make sense to me. I do not see the logic.

I am very interested in XML files. Please post them even if they look useless to you. -- | ,__o | _-\_<, http://www.gnu.franken.de/ke/ | (*)/'(*)

People being eager to try TEI and there being no official place to post TEI files, everybody has posted the files they have marked up in a different place. I have been working on my dialect, Jeroen on his and DP is cooking up another one. There is no central "clearing house" where we can see the other guys work. I don't say it would be impossible for me to obtain a glimpse of the TEI texts the folks at DP are working on, it would just be much easier if I could get them from the archive.

Personally, I try to stick as closely to TEILite as possible. I can add extentions to it, but then can easily produce XSLT to pull out those extentions before posting. I think a few of Marcello's extentions for PGTEI are not needed, as elements exists to encode the same information, or alternative mechanisms can be divised within TEILite -- but even if you stick to pure TEILite, you will need to agree on conventions, for example, I leave in quotation marks (as I have numerous old works that deal with these in a very irregular way, turning them to <q> and </q> would be difficult. Marcello leaves them out. We can fix an XSLT to re-supply them, and even an XSLT to supply them only if they are removed (given we agree on a standard way of documenting this fact) -- and that is what you need if you're working on a certain project using TEI -- a gentle intruduction and some guidelines. A few very nice ones are on the Net. If people wish, I can set up a website with TEI versions of _all_ my posted texts, both in my original master SGML, and converted to XML. Gives everybody something to experiment with. Jeroen.

Jim Tinsley wrote:

...

I feel Jim is raising artificial objections he knows we cannot overcome. If he doesn't want to learn TEI and he doesn't feel like proofing a TEI text in emacs, fine. But then, he should step aside and let other people do this work.

I find this very offensive.

I came home, and was reading happily enough through the threads until this.

I am sorry if I spoilt your evening and I apologize for that. I said "I feel" and that's the truth. Maybe it's just my fault.

these are the expectations I have had for XML as far back as I can remember, and they are expectations regularly assumed, if not met, by people who evangelize XML.

Some of your expectation cannot be met. Some would imply an enourmous expense of time on the developers part to save relatively little time on your part.

I "learned TEI" (not all of it, of course) with the hope of using it in PG, in late 2001/early 2002, and I marked up my first book in XML in February, 2002, which was long before I ever heard your name.

I have marked up 25 books, prose, lyrics and plays. And I transformed all of them successfully to HTML, TXT, PDF and PalmDoc. That was a year ago. I could have done more but I felt that it was better to go public with what I had, to get comments and suggestions from other people. I thought if PG posted some of those files I would get comments. Since then I have been waiting. I think I have done my part. My files are done better than many I see posted. Even if we had to fix them later, the philosophy of PG did at some point expressly allow the posting of preliminary files. I cannot see why this simple request should cause so much trouble and fear today. These are some of your expectations that you should reconsider:

I really no longer give any headroom at all to the approach "Post XML Now Because That Is The One True Way And We'll Figure Out How To Read It Later." If for no other reason, then because the most important part of the WW job is to check the texts before posting, and if we can't read it, we can't find the errors, and if we can't find the errors, we can't fix 'em.

You can read a TEI file in an editor. You can spell-check it. You can validate it. You can find the errors. The process is just a bit different from what you have now, and will always be until there crop up some native TEI readers.

That process must work for _all_ teixlite files, not just ones that are specially cooked, using constraints not specified within the chosen DTD. Here's where we hit the rocks today.

Impossible. There are things you cannot specify in a DTD but still must be followed to get a semantically correct file. (This holds for every XML application not just for PGTEI.) You always have to obey some extra rules besides validity. These are put down in the PGTEI guide.

The only things we must have -- both for our own internal practical purposes and for the use of future readers -- is that it should work reliably on _all_ texts that conform to the XML DTD chosen, be open source, and be cross-platform. A reader needs to be able to tweak the transform and re-run on her own desktop.

Same as above. The DTD is not strict enough (RelaxNG will be better, but it's still early). There will always be valid TEI files that do not transform to `correct' output files. I don't see why it is necessary for the conversion tools to run on everybodies desktop before we can start posting files. If the tools run on pglaf.org and gutenberg.org that is more than enough for a start. The tools can be fixed later. That won't make posted valid TEI files invalid. -- Marcello Perathoner webmaster@gutenberg.org

Two points to mention before you read my actual reply: 1) My apologies for the delay in responding. A configuration issue has caused all my email to bounce from pglaf.org's mail server for the past couple weeks, and only yesterday was I able to get it rectified. I will be posting many delayed replies for the next few days. 2) While the original email that I'm replying to here was written by Marcello, my replies are not directed at him personally, but rather, at everyone in the community. ---------------------------- Of the many assumptions being made on the list these days, here are three of the most erroneous -- and they all came in a single paragraph.

We don't need more discussion about whether TEI is the right language, I think we are all agreed on that.

Not everyone agrees with the use of TEI. I'm not even going to begin my arguments again -- there would no point. Some of us, such as myself, foresee issues, based on our own experiences, with using TEI (or varient) for PG work. I'm resigned to the fact that it may be the only way to get XML into PG at all, so I'll just deal with the issues on my own time at that point. Simply put, TEI is one of the most verbose markup vocabularies available, and using it for PG is going to turn off a LOT of people to XML. A simpler, more concise vocabulary would be less intimidating!

pgxml.org is dead

PGXML is not dead at all. Just like other XML stuff in PG, it's been on hiatus, mostly for two reasons: 1) lack of agreement in the community, and 2) lack of personal time to work on it. I will be meeting with the other co-founder of pgxml.org (Ben Crowder) in November, after which time, we hope to present a definitive plan for pgxml.org to the community.

and ZML is good for laughs.

You can laugh at ZML all you want, but from the examples and personal discussion with Bowerbird, I have learned that ZML is not at all what most people think it is. From the examples that I have seen, ZML is basically PG vanilla text format, but cleaned up and normalized. ---------------------------- The entire rest of this email is a rant, so please feel free to skip it. You have been given the choice! ---------------------------- Maybe if you learned to listen to other people, you'd not make such erroneous assumptions. Maybe, just maybe, other people do have a clue, and you aren't the only one that knows something. Yes, I have blocked Bowerbird from joining the PGXML list, but he is the _only_ person that I've blocked. The only reason for this is because of other people's reactions to him, not because of Bowerbird himself. While he can be irritating and annoying, Bowerbird does have a clue about some of the issues we have to deal with in PG work, particuarly in converting to other formats, etc. If you don't like his attitude, ignore his posts. You can at least try to extract the useful information that he does give from the flame wars they often come in. This way, you might actually LEARN something. More mud is slung on this list than on ANY other list that I'm subscribed to, but I have to admit, I'm only subscribed to about 200 active lists, so I may be missing the mud-slinging ones. If you don't like the way something is being done in PG, don't throw a hissy-fit. Get off your arse and do something about it, or sit down and shut up, and let other people do what they think should be done. I've made no secret of my personal opinions of PG: 1) the website is a disgrace 2) the archive is poorly organized 3) the catalog system is a hack job done by unqualified people 4) the PG text format is extremely disgusting 5) PG makes volunteers work uphill to get anything done 6) the lack of quality in our content offsets any gain from it Just because I have the opinions doesn't automatically mean I A) know what I'm talking about, or B) have an alternative solution. Some of the biggest technological innovations cames not from people who had a better idea, but from people who knew the current idea wasn't that good, and were open minded when the better idea came along. As a whole, I find that PG is not a very open-minded community. As a community, we reguarly discourage people from volunteers, mostly because we don't support them well. At times, not only do we not support them, but we actively, and publically, bash their skulls. We lie to the general public about PG on a regular basis. When posting ebooks, we ignore the wishes of the volunteers who made the texts. We don't even provide well-suited tools for the volunteers to use to improve PG, because, oh my god, maybe the tool isn't 100% open-source! Maybe the tool has been offered to PG on a perpetual right to use for PG status, but oh, lordy, that's just not good enough. We reguarly tell some of our hardest working volunteers that they are full of crap (more or less). True, it's usually a paragraph long description of what they are doing "wrong", but it's basically telling them their work is unwanted. Do you realize that out of over 300 librarians I've talked to personally, the first thing that came to mind for "ebooks" was the University of Virginia's eText Library? PG simply is not a mover or a shaker in the ebook world, regardless of the hocus pocus you might hear to the contrary. ---------------------------- And, to all these things I've said, I'm a victim of some, but a perpetrator of others. I'm just as guilty as you are. I have no soapbox, only a conscience and a desire to change things. ---------------------------- So, dangit all to heck, we don't have to be like this. PG _could_ be great. PG SHOULD be great. -- James

Oh geez... I'm going to ask Ben what happened. :-( Ok, so as far as I know pgxml.org doesn't belong to us anymore, but that doesn't make it dead, just the domain will have to change. -- James

-----Original Message----- From: gutvol-d-bounces@lists.pglaf.org [mailto:gutvol-d-bounces@lists.pglaf.org]On Behalf Of Marcello Perathoner Sent: Saturday, October 23, 2004 3:39 pm To: Project Gutenberg Volunteer Discussion Subject: Re: [gutvol-d] barriers to XML posting

James Linden wrote:

...

...

pgxml.org is dead

PGXML is not dead at all.

I stand corrected. Sometime ago I noticed the domain was expired, but somebody re-registered it just yesterday. But I gather it wasn't you. Who is this ?

...

$ whois pgxml.org Domain ID:D105038884-LROR Domain Name:PGXML.ORG Created On:22-Oct-2004 20:47:27 UTC Last Updated On:22-Oct-2004 20:47:48 UTC Expiration Date:22-Oct-2005 20:47:27 UTC Sponsoring Registrar:Go Daddy Software, Inc. (R91-LROR) Status:TRANSFER PROHIBITED Registrant ID:GODA-08593143 Registrant Name:Registration Private Registrant Organization:Domains by Proxy, Inc. Registrant Street1:15111 N Hayden Rd., Suite 160 Registrant Street2:PMB353 Registrant Street3: Registrant City:Scottsdale Registrant State/Province:Arizona Registrant Postal Code:85260 Registrant Country:US Registrant Phone:+1.4806242599 Registrant Phone Ext.: Registrant FAX: Registrant FAX Ext.: Registrant Email:PGXML.ORG@domainsbyproxy.com ...

-- Marcello Perathoner webmaster@gutenberg.org

_______________________________________________ gutvol-d mailing list gutvol-d@lists.pglaf.org http://lists.pglaf.org/listinfo.cgi/gutvol-d

The second is that while a more simple XML markup, like what you loosely described as PGXML, sounds wonderful on the surface ... it requires, once again, largely reinvented the wheel

"Reinventing the wheel" is often something to be avoided, but I'm not sure it's a compelling issue here. First, there are other models to use, e.g. XHTML. Second, the most important standard is XML itself. That's what enables an incredible variety of tools and platforms; the specific DTD is much less important. (In fact, XML's designers made sure it was useful even without a DTD.) Third, TEI was created for a very different world: scholarly publishing. If PG's markup was going to be done by paid experts, TEI would probably be the best choice. But I'm not convinced it's appropriate for a volunteer organization. XML can be much simpler than HTML, yet TEI is (IMHO) more complex not less. I just finished converting The Wonderful World of Oz to PGTEI. (I'll post it on Classicosm.com once I have a chance to write up my impressions.) During my learning process, I came across an interesting comparison of Shakespeare marked up using TEI and an "ad hoc" markup used by Jon Bosak (a key inventor of XML). Though the comparison was done by a TEI advocate, I think Jon's is a much better model for our purpose. http://www.tei-c.org.uk/Sample_Manuals/mueller-main.htm A very gentle introduction to the TEI (the comparison is near the end -- look for the garish background colors)

Granted, the very nature of XML makes converting from a home-grown markup to TEI a possibility, removing the need to convert would seem to be the wiser path.

The whole point of a master format is that PG is going to convert to other useful formats. If TEI is useful in and of itself, that can be just another conversion. -- Scott Practical Software Innovation (tm), http://ProductArchitect.com/

...

I've made no secret of my personal opinions of PG:

1) the website is a disgrace 3) the catalog system is a hack job done by unqualified people

That's what I've fixed in the last year.

Yes, I have to admit that you've been working VERY hard on the site, but in my own opinion, it is still not up to par. That's not really any fault of your's tho. I wouldn't call your new catalog system "fixed", but it IS better than the old CGI one. :-)

...

4) the PG text format is extremely disgusting

That's what I tried to fix. But ran against 5)

...

5) PG makes volunteers work uphill to get anything done

Hey, I'm with you there. :-| -- James