let's talk about the issue of putting scans online.
the discussions on scanning file-naming conventions
and the resolution at which we "should" be scanning
have sidetracked the greater thread into a tar-pit...
in an attempt to pull it out, i offer this post,
which i wrote after having examined a couple
of the e-texts which currently offer pagescans.
perhaps this re-entry into the world of reality will
remind us that all of this _does_ bring up issues,
even if they are not the ones we're agonizing over.
since i've been writing this over the course of
several days, some of it is focused on basics
that you might think we have already covered,
but try to read it with an open mind if you can...
the fact of the matter is, unlike the early days,
we now have scans for a very high percentage
of the e-texts that are put into the p.g. library.
they might not meet every one of the demands
that some of us would like to impose on them,
but nonetheless, we _do_ have the pagescans...
and, with diskspace getting cheaper all the time,
and fat pipes becoming more and more common,
we can start entertaining the idea of making these
pagescans available on a public basis. and indeed,
the "official" policy is that scans can be included...
especially from the standpoint of _error-correction_,
having online scans could make a _huge_ contribution.
but even if we decide to hold off on this for a while,
it's still a good idea to know what we need to know
when we _do_ go ahead and start putting scans up.
so let's examine this question from a _practical_
standpoint, rather than some "idealistic" one...
and the best way to do that is to get real...
***
i looked at four english e-texts from the library
where the pagescans are already included, and
will first make some overall comments about them.
as i said a few days back, these are case-studies
in how _not_ to do this. they really are that bad.
here are some of the problems.
some of the scan files are bad; they will not display.
for instance, 2 of the 4 images in #14116 are bad.
that's not a very good percentage, i would consider.
3 of the 4 e-texts contained at least one bad scan.
the naming convention is also flawed, badly flawed.
(i guess we cannot get away from this completely.)
a naming convention that uses filenames like
"001.tif", "002.tif", etc. is completely useless.
when you have 17,000 such e-texts, when some
go astray, as they inevitably will at some time,
you do _not_ want to have to open up each
"001.tif" in order to see what it _really_ is.
if you've ever experienced that, you know that
it drives you nuts, and you find a way to fix it.
laying a "system" like this on the incompetents
in the general public is a recipe for huge snafu.
the name _has_to_ reflect the book itself,
so there needs to be a _rootname_ that is
unique to each book prefixing the filename.
that allows different e-texts to live in the
same folder, and that can come in handy.
since the "rootname" that p.g. has come to
decide upon is the e-text-number, use that.
so the name for the first scan in #14116
should be something like "14116-001.tif"
(there is one school of thought that says
that you should only zero-pad the numbers
as far out as you need for the particular book.
thus, since there are only 4 pages in this e-text
-- it's one issue of a serial -- that would mean
_no_ zero-padding, so the name of the first file
would be "14116-1.tif" if you follow this school.
your tools must know how to resolve the issue to
find the right file no matter which way you do it.)
again, with 4 scans in this e-text, and 2 of them bad,
this e-text does not serve as an auspicious beginning.
***
ok, after #14116, let's look at #14040. at the outset,
it is worth noting here that the quality of these scans
is not very good. so i am uncertain _why_ this project
was chosen as one where the raw images are included.
these are not the type of scans you will hold up proudly.
you might be willing to show them if someone insisted
that they had a reason that they needed to see them.
but they're not good enough to parade 'em as examples.
aside from the useless "001.tif" file-naming problem again,
there are some other file-naming problems with this e-text.
first of all, the filenames do not reflect the pages that are
contained within them. as above, you don't want to have to
_open_ a file to know the number of the page pictured inside.
you want that number reflected in the name of the file itself,
to the degree that it is possible. (and it almost always is.)
as it is, a few example files from #14040 are as follows:
> 003.tif = page 7
> 004.tif = page 9
> 005.tif = page 11
> 006.tif = page 12
so, we _could_ rename the files as follows:
> rename 003.tif as 14040-007.tif = page 7
> rename 004.tif as 14040-009.tif = page 9
> rename 005.tif as 14040-011.tif = page 11
> rename 006.tif as 14040-012.tif = page 12
looking at these scans shows they are front matter,
so we can probably safely assume pages 8 and 10
were blank left-hand pages. but we don't actually
_know_ that now, do we? and we want to _know_;
we don't want to have to _assume_ any such things,
especially if we need to open up and examine the scans
in order to be able to be confident about our assumptions.
so this solution only makes us wonder about 008.tif,
and 010.tif. what was up with those pages/files?
are they blank pages? or missing pages? or what?
the upshot is that blank pages have to be included,
because we need to have them as _placeholders_
which tell us that we do indeed have all of the pages.
we want our checker-apps to be able to run through
all of the filenames in a folder and _confirm_that
"yes, all pages seem to be present and accounted for",
and to do that, we need to have files for blank pages.
this also addresses a less subtle problem, which is that
-- for some particular usages -- apps will need to know
whether a page is a _left-hand_page_ in the p-book or
a _right-hand_page_. when the collection of filenames
is fully in-sequence with the pages as they were bound,
with a separate file for each and every recto and verso,
we can know -- from the odd/even nature of the file's
position in that enforced-linearity sequence-number --
whether that page was a left-hand or a right-hand page.
in the bad example above, the linkage has been corrupted.
003.tif is a right-hand page, but so are 004.tif and 005.tif.
our assumption would be that 004.tif was a left-hand page.
if we want to print the scans to "recreate" the paper-book --
which will be one of the most-common uses scans will serve
-- an absence of synchronicity like this one simply will not do.
just to be thorough in my treatment, i'll remark that
this recto/verso distinction also has implications when
you have an image that's inserted between two pages.
you have to include a blank-page scan after that, too,
for its _verso_, to keep the sequence of files consistent.
you might remember that i pointed this out to jon noring
when it came up in his "my antonia" project a while back.
marcello's illustration-file in his djvu has this problem too.
he inserted a graphic named "72-image.tif" within the files,
but didn't have the corresponding blank-verso file as well.
now whether a person will actually _scan_ these blank pages
is an open question. i found that trying to skip the blank pages
broke my rhythmic scanning pattern, so it was faster and easier
for me to just scan them. but other people might be different.
if they are, we can deal with their output later, with no trouble;
it's really not a big deal to rename the files appropriately and to
insert the blank-page scans wherever they happen to be needed.
(you just have to have a tool that's smart enough to do the job.)
one more thing about this "recreating the paper-book" thought.
if the files are named wisely, sorting their names will give you
the order in which they should be printed to mimic the p-book.
so there's no need for an "index file" or that kind of nonsense.
sort-order is the very type of easy-to-follow no-ambiguity rule
that program-designers like. it also means that people can use
a slide-show tool to cruise through the pagescans if they want.
note that a sort-order rule conflicts with marcello's suggestion
that scans of the covers should be named as follows:
> c0000.tif = front cover
> c0002.tif = inside front cover
> c0003.tif = inside back cover
> c0004.tif = outside back cover
> c0005.tif = spine
according to the binding-creates-a-specific-linearity model
which we then emulate through judicious naming of the files,
names for the back-cover scans should sort to the _bottom_.
***
um, we're not done with 14040 yet; there are
some other serious problems with this scan-set.
pages 30 and 31 were scanned twice, whereas
pages 32 and 33 are missing. if you're looking,
this sequence is of files 024.tif through 027.tif.
(and here you can see how confusing it becomes
when the filenames are not the page-numbers.)
this type of error is a not-uncommon one when scanning, to be sure.
duplicate spreads, or missing spreads, or -- as in this case -- a combo,
are easy mistakes to make in the course of scanning hundreds of pages.
after all, it is humans doing this scanning. and to err is human, right?
(but sheet-fed scanners _also_ make these mistakes, when they jam,
or misfeed, or whatever. so it's not just humans; machines err too.)
but even (especially!) if a mistake is common, it _should_ be the case
that quality-control checks would have located and fixed this mistake,
_long_ before this scan-set was ever released out to the general public.
indeed, this type of mistake should _always_ be caught -- and fixed! --
before the person doing the scanning even returns the book to the shelf.
it takes 20 times longer to fix this problem if you don't do it right away.
the fact that the error is still sitting there, easy for me and anyone else
to check and see, indicates that quality-control needs to be improved...
finally, the "read-me" file for these scans says:
> The page images of thie book
> are shown in the TIFF files.
it might be fairly petty to note this, but a typo on a word like "this"
is pretty embarrassing for a "literary foundation" like p.g., is it not?
***
ok, on to marcello's djvu example, for text #12973.
(note the slight expansion of the analysis, from just the scans themselves
to now include their consolidation within the djvu file, an additional step.)
first, as i had noted in a earlier post, the page-numbers on the _djvu_file_
are _out-of-sync_ with the page-numbers on the actual pages in that file.
but unlike the examples above, that's _not_ because the pagescan files
are mis-named. in this case, those pagescan files are named _correctly_.
(well, with just 2 exceptions.)
what caused the problem, however, is that one of the pagescan files
is named "000.tif", and djvu does not have the concept of a page zero.
the first page it encounters is page 1, and then each page after that is
incremented by 1. to put it another way, djvu does its own numbering.
(this flaw is not unprecedented with viewer-tools; acrobat is the same.
you'd think that any program that purports to deal with electronic-books
would have -- at the outset -- taken into account that forward matter is
often numbered with roman numerals, and that "page 1" of a book will
require resetting the counter. but i guess that's expecting too much.)
so that 000.tif -- a picture of the person who is the subject of the book,
which the .html-version graphic-name enlightens us is the frontispiece --
throws off the numbering from the very beginning of the djvu file. oops.
ironically, there is a "blank page" on page 4/5, which that frontispiece
could replace nicely, and that would leave the page-numbering correct.
if it provides us with accurate pagenumbers throughout the e-book, it is
worth it to shuffle some pages of the front-matter around to achieve it.
(even if it upsets the "replicate the p-book" objective mentioned above.)
and in this case, that shuffle would indeed give us correct pagenumbers.
at least until we hit the picture on page 73 (or so, it's 072-image.jpg),
which -- as a separate file -- threw off marcello's djvu page-numbering
by an _additional_ increment of 1, for a combined offset now of _2_,
which means the offset is not even continuous throughout this djvu file.
it's an offset of 1 up until page 73 (or so), then an offset of 2 after that.
how can an ordinary person keep stuff like this straight?
while it is not altogether clear _where_ this picture was in the p-book,
to fix this, this picture could have been placed at the bottom of page 65,
a half-page at the end of the previous chapter. and thus, the relationship
between the djvu computed page-number and the p-book page-numbers
in this scan-set _could_ be made to be totally consistent, if we wanted,
by incorporating both of these fixes. it is often the cause that you can
achieve this harmony. if you just do some work, you can make it work.
but there are also some cases where you just cannot make it work out.
if a p-book has 20 pages of front matter, and the first page of the book
is numbered "3", there's no way you can do the required squeezing, so
-- for those tools that do their own page-numbering starting at zero --
you're just not going to be able to make it work right. i recommend you
lobby the programmers of those apps to make 'em more sophisticated,
or else you can just throw them out. (the tools, not the programmers.)
:+)
in the meantime, _my_ tools will be programmed to do things correctly...
(because having correct page-numbers for the body-text is so important,
one way to make the best of a bad situation is to move the "front matter"
to the end of the regular text, with a note in its original location that
tells
users where they can find it, and why it was moved for their convenience.
this throws the numbering off for the front-matter, but that's less serious.
again, it would be much nicer if the tools just accommodated this situation,
since it is extremely common. but sometimes you do what you have to do.)
***
and now back to the analysis of the scan-set and the djvu of #12973...
concentrating now on the pagescans themselves, rather than the djvu,
many scans are badly skewed. in particular, the left-hand pages are
tilted one way, and the right-hand pages the other way, which will be
a familiar sight to people who have experience with looking at scans,
since it reflects a physical reality of putting a bound book on a scanner.
sadly, as you page through the scans, this back-and-forth tilting gives the
impression of being on a ship. it can even make you a bit seasick...
:+)
so this introduces the general topic of the _clean-up_ of the scans...
and the first way that scans need to be cleaned-up is to be _straightened_.
when you're scanning hundreds of pages, almost all of them will be crooked,
to one degree or another, no matter how meticulous you are trying to be.
this skew of the pages can make them difficult to read if it is bad enough.
and even if it's only a very subtle skew, it will bother readers'
subconscious.
it's also worth noting that skewed pages give particularly poor o.c.r.
results.
so if you are going to deskew a page sooner or later (and -- if you want to
make 'em public -- you really have to), then you might as well deskew them
_before_ you o.c.r. them, and save yourself some time correcting scannos.
(some o.c.r. programs even deskew the image before they do the recognition.
i don't know if they save the deskewed image, though, which is what we want.)
the people over at d.p. have finally learned this,
at least when it comes to a really bad skew,
of the type that is the case in almost all of the
scans in this book. so we won't see pagescans
that are this badly skewed, not from books that
were done recently. however, i'm not sure that
the d.p. people have learned that even a _little_
skew can adversely impact their o.c.r. results,
so i don't think they routinely deskew all scans.
but maybe they've changed. if not, they should.
next, the size of the images in this scan-set has not been standardized.
in particular, the partial-pages that are found at the end of each chapter
are smaller, and thus -- when using the "fit-to-page" view option in djvu
(which is the only realistic choice in most cases, providing the page can
actually be read at that scaling, a problem with any non-reflowing format)
-- it gets blown up unnecessarily large relative to the surrounding pages.
except in very rare cases, all pagescans of a book should be the same size.
in addition, positioning of the text on the scans should be _regularized_.
that is, the upper-left point of the text should fall at the exact same point
on every scan. even those end-chapter pages that are half-full should
have their _top_left_ at the same point, even though their bottom will
of course fall at a different place. (and, as noted above, they should
still
be the exact same size -- in width and height -- as all of the other pages.)
although you might not see why at first, regularization can also aid o.c.r.
that's because regularization allows you to set "zones" on the pagescan,
one time for all of the pages, and zoning keeps the o.c.r. results straight.
for instance, the zone that contains the running-head will be recognized as
being independent of the body-text itself, so their text is not joined
together.
another example is that zoning can rule out stray marks in the margin from
being recognized, and thus keep junk from intruding into the o.c.r. results.
***
the last of the four e-texts i looked at was #14100. the problems with
this set of scans were the same ones that have been discussed above,
including a few very skewed scans, and a lack of regularized positioning.
but otherwise, though, the quality of the scans was relatively good, and
none of the 27 .tiff images were corrupted -- they all displayed just fine.
***
so, there's a look at some actual e-text pagescans in the library.
(there are a few more, but they were not in english, so i passed.)
as it shows, the current examples of pagescans in the library
are not very good examples at all, except as _bad_ examples.
and if an analysis of _4_ e-texts results in a post this long,
imagine the huge mess that will result from _400_ e-texts,
if the practices are not buttoned down before we even start.
(we won't even think about _4,000_, or _4-times-4,000_.)
some of the flaws -- skewed scans, missing pages, etc. --
are readily apparent. others, like the filenaming glitches,
may be less obvious to the casual observer, but they still
stick out like sore thumbs to those of us who are trying to
build tools to extend people's power in using the library...
it is also of importance to note that some of these steps
-- most notably deskewing and regularized text positioning --
are ones that can substantially improve the o.c.r. accuracy,
which is a bonus that we probably cannot afford to pass up.
if you're gonna do scan-cleanup sooner or later, do it sooner!
other of my suggestions here, such as filenaming conventions,
would also help the d.p. workflow, and i do believe this strongly.
but if they want to be stubborn and do it some other way, fine.
i'll just have to make corrections to what they do after-the-fact.
***
anyway, there's my analysis for now.
hopefully, instead of dealing with red herrings, these real issues
will be addressed when project gutenberg makes the decision to
move forward in earnest on the task of making scans available...
that's all for now...
-bowerbird
