[gutvol-d] Re: Fwd: Programmatic fetching books from Gutenberg

29 Jul 2009

      On Tue, Jul 28, 2009 at 06:24:08PM -0600, Joey Smith wrote:
...
On Tue, Jul 28, 2009 at 07:33:06AM -0700, Greg Newby wrote:
...
Somebody would need to write the software :)
Zipping an mp3 is not a winning strategy: they really don't
compress much, if at all.
Putting multiple mp3 files for a single eBook in one file,
on the fly, would be a great move - making it easier to 
download a group of files.
A more general approach would be to let visitors to www.gutenberg.org
put their selected files (including those generated on-the-fly)
on a bookshelf (i.e., shopping cart), then download in one big file,
or several small ones.
This would involve some fairly significant additions to the
current PHP-based back-end at www.gutenberg.org, but is certainly
not a huge technical feat.
  -- Greg
Where can one find the code for the "current PHP-based back-end at www.gutenberg.org"
to begin doing looking into how feasible this would be?
Thanks for your interest :)

It isn't bundled up for download anywhere.  We'll probably need to wait
for Marcello's return from vacation to provide details on how to add
components like this.  The current system is modular & (I think)
well-organized, but complex...including lots of stuff that readers never
see (such as the cataloger interface and various programs that add new
files).  Plus, as you know, there is a lot of stuff that is in the
Wiki, rather than PHP.  The Wiki might be where new features could
be added, or there might be modules "out there" that could make it easier. 

I did grab catalog/world/bibrec.php , where bibrecs like this are
made:
  http://www.gutenberg.org/etext/11

It is below.  This should give you an idea how where various things are
tied in from the database, the on-disk cached records, and stuff that is
generated on the fly.  The various .phh files it references (which cascade
to include a whole bunch of stuff) are mostly for presentation (html
and css), not functionality.

A bookshelf/shopping cart would probably be a brand new set of files,
with just a little overlap with the existing php.  It would need to
access the database, and presumably would need a table or two to keep
track of bookshelf users & entries.  (Maybe a separate database...maybe
part of the Wiki instead of a standalone set of PHP programs.)  Cookies,
or something similar, could be used to track user sessions and their
bookshelves/shopping carts/whatever, and add an entry to various pages
at www.gutenberg.org for them to access it (sort of like a regular
ecommerce site).

-------- bibrec.php
<?php

include_once ("pgcat.phh");

$cli = php_sapi_name () == "cli";
if ($cli) {
  $fk_books = intval ($_SERVER['argv'][1]);
} else {
  $devserver = preg_match ("/www-dev/", $_SERVER['HTTP_HOST']);
  if ($devserver) {
    nocache ();
  }
  getint ("fk_books");
}

$db = $config->db ();

$keywords     = array ();
$frontispiece = null;
$category     = 0;
$newfilesys   = false;
$help         = "/wiki/Gutenberg:Help_on_Bibliographic_Record_Page";
$helpicon     = "<img src=\"/pics/help.png\" class=\"helpicon\" alt=\"[help]\"$config->endtag>";

$db->exec ("select * from mn_books_categories where fk_books = $fk_books order by fk_categories");
if ($db->FirstRow ()) {
  $category = $db->get ("fk_categories", SQLINT);
}

$friendlytitle = friendlytitle ($fk_books, 80);
$config->description = htmlspecialchars ("Download the free {$category_descriptions[$category]}: $friendlytitle");

for ($i = 0; $i < 26; ++$i) {
  $base32[sprintf ("%05b", $i)] = chr (0x41 + $i);
}
for ($i = 26; $i < 32; ++$i) {
  $base32[sprintf ("%05b", $i)] = chr (0x32 + $i - 26);
}

// find best file for recode facility

class recode_candidate {
  function recode_candidate () {
    $this->score = 0;
    $this->fk_files = null;
    $this->filename = null;
    $this->encoding = null;
    $this->type     = null;
  }
}

function find_recode_candidate ($fk_books) {
  global $db;

  $candidate = new recode_candidate ();

  $db->exec ("select pk, filename, fk_encodings from files " . 
             "where fk_books = $fk_books and fk_filetypes = 'txt' " . 
             "and fk_compressions = 'none' and diskstatus = 0 and obsoleted = 0");

  if ($db->FirstRow ()) {
    do {
      $tmp = new recode_candidate ();
      $tmp->fk_files = $db->get ("pk", SQLINT);
      $tmp->filename = $db->get ("filename", SQLCHAR);
      $tmp->encoding = $db->get ("fk_encodings", SQLCHAR);

      if ((!isset ($tmp->encoding) || $tmp->encoding == "us-ascii")) {
        $tmp->score = 1;
        $tmp->encoding = "ASCII";
      }
      if ($tmp->encoding == "big5") {
        $tmp->score = 2;
        $tmp->encoding = "BIG-5";
      }
      if ($tmp->encoding == "euc-kr") {
        $tmp->score = 2;
        $tmp->encoding = "EUC-KR";
      }
      if ($tmp->encoding == "Shift_JIS") {
        $tmp->score = 2;
        $tmp->encoding = "SHIFT-JIS";
      }
      if (!strncmp ($tmp->encoding, "iso-", 4)) {
        $tmp->score = 3;
      }
      if (!strncmp ($tmp->encoding, "windows-", 8)) {
        $tmp->score = 4;
      }
      if ($tmp->encoding == "utf-8") {
        $tmp->score = 5;
        $tmp->encoding = "UTF-8";
      }

      if ($tmp->score > $candidate->score) {
        $candidate = $tmp;
      }
    } while ($db->NextRow ());
  }

  return $candidate;
}

function find_plucker_candidate ($fk_books) {
  global $db;

  $candidate = new recode_candidate ();

  $db->exec ("select pk, filename, fk_encodings, fk_filetypes from files " . 
             "where fk_books = $fk_books and (fk_filetypes = 'txt' or fk_filetypes = 'html')" . 
             "and fk_compressions = 'none' and diskstatus = 0 and obsoleted = 0");

  if ($db->FirstRow ()) {
    do {
      $tmp = new recode_candidate ();
      $tmp->fk_files = $db->get ("pk", SQLINT);
      $tmp->filename = $db->get ("filename", SQLCHAR);
      $tmp->encoding = $db->get ("fk_encodings", SQLCHAR);
      $tmp->type     = $db->get ("fk_filetypes", SQLCHAR);

      if ((!isset ($tmp->encoding) || $tmp->encoding == "us-ascii")) {
        $tmp->score = 1;
      }
      if ($tmp->encoding == "iso-8859-1") {
        $tmp->score = 2;
      }
      /* if ($tmp->encoding == "windows-1252") {
        $tmp->score = 3;
      } */
      if ($tmp->type == "html") {
        $tmp->score = 4;
      }
      if ($tmp->score > $candidate->score) {
        $candidate = $tmp;
      }
    } while ($db->NextRow ());
  }

  return $candidate;
}

function base32_encode ($in) {
  global $base32;

  $bits = "";

  $in = @pack ("H*", $in);
  $len = strlen ($in);
  for ($i = 0; $i < $len; $i++) {
    $bits .= sprintf ("%08b", ord ($in{$i}));
  }

  if ($mod = strlen ($bits) % 5) {
    $bits .= str_repeat ("0", 5 - $mod);
  }

  return strtr ($bits, $base32);
}

class DownloadColumn extends dbtSimpleColumn {
  function DownloadColumn () {
    global $help, $helpicon;
    parent::dbtSimpleColumn (null, "Download Links <a href=\"$help#Download_Links\" title=\"Explain Download Links.\">$helpicon</a>", "pgdbfilesdownload");
  }
  function Data ($db) {
    global $config, $friendlytitle, $fk_books, $newfsbasedir;

    $filename = $db->get ("filename", SQLCHAR);

    $extension = "";
    if (preg_match ("/(\.[^.]+)$/", $filename, $matches)) {
      $extension = $matches[1];
    }

    $dir = etext2dir ($fk_books);
    if (preg_match ("!^$dir!", $filename)) {
      $symlink = preg_replace ("!^$dir!", $newfsbasedir, $filename);
    } else {
      $symlink = "$config->downloadbase/$filename";
    }

    $links = array ();

    $links[] = "<a href=\"$symlink\" title=\"Download from ibiblio.org.\"><span style=\"font-weight: bold\">main site</span></a>";
    $links[] = "<a href=\"$config->world/mirror-redirect?file=$filename\" title=\"Download from mirror site.\" rel=\"nofollow\">mirror sites</a>";

    $sha1 = base32_encode ($db->get ("sha1hash", SQLCHAR));
    $tt   = base32_encode ($db->get ("tigertreehash", SQLCHAR));

    $links[] = "<a href=\"magnet:?xt=urn:sha1:$sha1" .
      "&xt=urn:kzhash:" . $db->get ("kzhash", SQLCHAR) .
      "&xt=urn:ed2k:"   . $db->get ("ed2khash", SQLCHAR) .
      "&xt=urn:bitprint:$sha1.$tt" .
      "&xs=http://$config->domain$symlink" . 
      "&dn=" . urlencode ("$friendlytitle$extension") .
      "\" title=\"Magnetlink to download from P2P network.\">P2P</a>";

    return "<td class=\"pgdbfilesdownload\">" . join (" ", $links) . "</td>";
  }
}

$array = array ();

$db->exec ("select * from books where pk = $fk_books;");

if (!$db->FirstRow ()) {
  error_msg ("No etext no. $fk_books.");
}

$release_date = $db->get ("release_date");
$copyrighted  = $db->get ("copyrighted") ? 
                "Copyrighted. You may download this ebook but you may be limited in other uses. Check the license inside the ebook." : 
                "Not copyrighted in the United States. If you live elsewhere check the laws of your country before downloading this ebook.";

$db->exec (
"select * from authors, roles, mn_books_authors 
where mn_books_authors.fk_books = $fk_books
and authors.pk = mn_books_authors.fk_authors
and roles.pk = mn_books_authors.fk_roles order by role, author"
);
$db->calcfields ["c_author"] = new CalcFieldAuthorDate ();

if ($db->FirstRow ()) {
  do {
    $pk   = $db->get ("fk_authors", SQLINT);
    $name = $db->get ("c_author",   SQLCHAR);
    $role = htmlspecialchars ($db->get ("role",     SQLCHAR));
    $array [] = preg_replace ("/ /", " ", $role);
    $array [] = "<a href=\"/browse/authors/" . find_browse_page ($name) . "#a$pk\">$name</a>";
    $keywords [] = htmlspecialchars ($db->get ("author", SQLCHAR));
  } while ($db->NextRow ());
}

$db->exec ("select attributes.*, attriblist.name, attriblist.caption from attributes, attriblist " . 
           "where attributes.fk_books = $fk_books and " . 
           "attributes.fk_attriblist = attriblist.pk " . 
           "order by attriblist.name;");

if ($db->FirstRow ()) {
  do {
    $note    = htmlspecialchars ($db->get ("text", SQLCHAR));
    $caption = htmlspecialchars ($db->get ("caption", SQLCHAR));
    $note    = preg_replace ("/\n/", "<br$config->endtag>", $note);

    if ($caption) {
      $name  = $db->get ("name", SQLCHAR);
      switch (intval ($name)) {
      case 901:
        $note = "<a href=\"$note?nocount\"><img src=\"$note?nocount\" title=\"$caption\" alt=\"$caption\" $config->endtag></a>";
        break;
      case 902:
      case 903:
        $note = "<a href=\"$note?nocount\">$caption</a>";
        break;
      case 10:
	$note = "$note <img src=\"/pics/link.png\" alt=\"\" $config->endtag> <a href=\"http://lccn.loc.gov/$note\" title=\"Look up this book in the Library of Congress catalog.\">LoC catalog record</a>";
	break;
      default:
	$note = strip_marc_subfields ($note);
	if (substr ($name, 0, 1) == '5') {
	  $patterns = array ("/http:\/\/\S+/", "/#(\d+)/");
	  $replaces = array ("<a href=\"$0\">$0</a>", "<a href=\"/ebooks/$1\">$0</a>");
	  $note = preg_replace ($patterns, $replaces, $note);
	}
      }
      $array [] = preg_replace ("/ /", " ", $caption);
      $array [] = $note;
    }
  } while ($db->NextRow ());
}

$db->exec ("select * from langs, mn_books_langs 
where langs.pk = mn_books_langs.fk_langs
and mn_books_langs.fk_books = $fk_books;"
);

if ($db->FirstRow ()) {
  do {
    $pk   = $db->get ("pk", SQLCHAR);
    $lang = htmlspecialchars ($db->get ("lang", SQLCHAR));
    $array [] = "Language";
    if ($pk != 'en') {
      $array [] = "<a href=\"/browse/languages/$pk\">$lang</a>";
    } else {
      $array [] = $lang;
    }
  } while ($db->NextRow ());
}

$db->exec ("select * from loccs, mn_books_loccs 
where loccs.pk = mn_books_loccs.fk_loccs
and mn_books_loccs.fk_books = $fk_books;"
);

if ($db->FirstRow ()) {
  do {
    $pk   = $db->get ("pk", SQLCHAR);
    $pkl  = strtolower ($pk);
    $locc = htmlspecialchars ($db->get ("locc", SQLCHAR));
    $array [] = "LoC Class";
    $array [] = "<a href=\"/browse/loccs/$pkl\">$pk: $locc</a>";
    $keywords [] = $locc;
  } while ($db->NextRow ());
}

$db->exec ("select * from subjects, mn_books_subjects 
where subjects.pk = mn_books_subjects.fk_subjects
and mn_books_subjects.fk_books = $fk_books;"
);

if ($db->FirstRow ()) {
  do {
    $subject = htmlspecialchars ($db->get ("subject", SQLCHAR));
    // $url = urlencode ($subject);
    $array [] = "Subject";
    //    $array [] = "<a href=\"$config->world/results?subject=$url\">$subject</a>";
    $array [] = $subject;
    $keywords [] = $subject;
  } while ($db->NextRow ());
}

$db->exec ("select * from categories, mn_books_categories 
where categories.pk = mn_books_categories.fk_categories 
and mn_books_categories.fk_books = $fk_books;");

if ($db->FirstRow ()) {
  do {
    $pk       = $db->get ("pk", SQLINT);
    $category = $db->get ("category", SQLCHAR);
    $array [] = "Category";
    $array [] = "<a href=\"/browse/categories/$pk\">$category</a>";
  } while ($db->NextRow ());
}

$array [] = "EText-No.";
$array [] = $fk_books;
$array [] = "Release Date";
$array [] = $release_date;
$array [] = "Copyright Status";
$array [] = $copyrighted;

$db->exec ("select count (*) as cnt from reviews.reviews where fk_books = $fk_books");
if (($cnt = $db->get ("cnt", SQLINT)) > 0) {
  $s = ($cnt == 1) ? "is a review" : "are $cnt reviews";
  $array [] = "Reviews";
  $array [] = "<a href=\"$config->world/reviews?fk_books=$fk_books\">There $s of this book available.</a>";
}

$newfsbasedir = "$config->files/$fk_books/";
$db->exec ("select filename from files where fk_books = $fk_books and filename ~ '^[1-9]/'");
if ($db->FirstRow ()) {
  $newfilesys = true;
  $array [] = "Base Directory";
  $array [] = "<a href=\"$newfsbasedir\">$newfsbasedir</a>";
}

for ($i = 0; $i < count ($keywords); $i++) {
  $keywords[$i] = preg_replace ("/,\s*/", " ", $keywords[$i]);
}
$config->keywords = htmlspecialchars (join (", ", $keywords)) . ", $config->keywords";

$recode_candidate  = find_recode_candidate  ($fk_books);
$plucker_candidate = find_plucker_candidate ($fk_books);
$offer_recode  = $recode_candidate->score  > 0;
$offer_plucker = $plucker_candidate->score > 0;

///////////////////////////////////////////////////////////////////////////////
// start page output

pageheader (htmlspecialchars ($friendlytitle));

$manubar = array ();

$menubar[] = "<a href=\"$help\" title=\"Explain this page.\" rel=\"Help\">Help</a>";

if ($offer_recode) {
  $menubar[] = "<a href=\"$config->world/readfile?fk_files=$recode_candidate->fk_files\" title=\"Read this book online.\"rel=\"nofollow\">Read online</a>";
}

p (join (" — ", $menubar));

echo ("<div class=\"pgdbdata\">\n\n");

$table = new BibrecTable ();
$table->summary = "Bibliographic data of author and book.";
$table->toprows = $array;
$table->PrintTable (null, "Bibliographic Record <a href=\"$help#Table:_Bibliographic_Record\" title=\"Explain this table.\">$helpicon</a>");

echo ("</div>\n\n");

$db->exec ("select filetype, sortorder, compression, " . 
           "case files.fk_filetypes when 'txt' then fk_encodings when 'mp3' then fk_encodings else null end as fk_encodings, " . 
	   "edition, filename, filesize, sha1hash, kzhash, tigertreehash, ed2khash " . 
	   "from files " . 
	   "left join filetypes on files.fk_filetypes = filetypes.pk " . 
	   "left join compressions on files.fk_compressions = compressions.pk " .
	   "where fk_books = $fk_books and obsoleted = 0 and diskstatus = 0 " . 
	   "order by edition desc, sortorder, filetype, fk_encodings, compression, filename;");
$db->calcfields ["c_hrsize"] = new CalcFieldHRSize ();

echo ("<div class=\"pgdbfiles\">\n\n");

echo ("<h2>Download this ebook for free</h2>\n\n");

class FilesTable extends ListTable {
  function FilesTable () {
    global $newfilesys, $offer_recode, $help, $helpicon;
    if (!$newfilesys) {
      $this->AddSimpleColumn ("edition",      "Edition",     "narrow pgdbfilesedition");
    }
    $footnote = ($offer_recode) ? " \xC2\xB9" : "";
    $this->AddSimpleColumn ("filetype",     "Format <a href=\"$help#Format\" title=\"Explain Format.\">$helpicon</a>", "pgdbfilesformat");
    $this->AddSimpleColumn ("fk_encodings", "Encoding$footnote <a href=\"$help#Encoding\" title=\"Explain Encoding.\">$helpicon</a>",    "pgdbfilesencoding");
    $this->AddSimpleColumn ("compression",  "Compression <a href=\"$help#Compression\" title=\"Explain Compression.\">$helpicon</a>", "pgdbfilescompression");
    $this->AddSimpleColumn ("c_hrsize",     "Size",        "right narrow pgdbfilessize");
    $this->AddColumnObject (new DownloadColumn ());

    $this->limit = -1;
  }
}

$array = array ();

function epub_file ($fk_books) {
  return "/cache/epub/$fk_books/pg$fk_books.epub";
}

function epub_images_file ($fk_books) {
  return "/cache/epub/$fk_books/pg${fk_books}-images.epub";
}

function mobi_file ($fk_books) {
  return "/cache/epub/$fk_books/pg$fk_books.mobi";
}

function mobi_images_file ($fk_books) {
  return "/cache/epub/$fk_books/pg${fk_books}-images.mobi";
}

$epub        = epub_file ($fk_books);
$epub_images = epub_images_file ($fk_books);
$mobi        = mobi_file ($fk_books);
$mobi_images = mobi_images_file ($fk_books);

// epub stuff

if (is_readable ("$config->documentroot$epub") && filesize ("$config->documentroot$epub") > 1024) {
  if (!$newfilesys) {
    $array [] = "";
  }
  $array [] = "EPUB (experimental) <a href=\"$help#EPUB\" title=\"Explain EPUB.\">$helpicon</a>";
  $array [] = "";
  $array [] = "";
  $array [] = human_readable_size (filesize ("$config->documentroot$epub"));
  $array [] = "<a href=\"$epub\" title=\"Download from ibiblio.org.\"><span style=\"font-weight: bold\">main site</span></a>";
}

if (is_readable ("$config->documentroot$epub_images") && filesize ("$config->documentroot$epub_images") > 1024) {
  if (!$newfilesys) {
    $array [] = "";
  }
  $array [] = "EPUB with images (experimental) <a href=\"$help#EPUB\" title=\"Explain EPUB.\">$helpicon</a>";
  $array [] = "";
  $array [] = "";
  $array [] = human_readable_size (filesize ("$config->documentroot$epub_images"));
  $array [] = "<a href=\"$epub_images\" title=\"Download from ibiblio.org.\"><span style=\"font-weight: bold\">main site</span></a>";
}

// mobi stuff

if (is_readable ("$config->documentroot$mobi") && filesize ("$config->documentroot$mobi") > 1024) {
  if (!$newfilesys) {
    $array [] = "";
  }
  $array [] = "MOBI (experimental) <a href=\"$help#MOBI\" title=\"Explain MOBI.\">$helpicon</a>";
  $array [] = "";
  $array [] = "";
  $array [] = human_readable_size (filesize ("$config->documentroot$mobi"));
  $array [] = "<a href=\"$mobi\" title=\"Download from ibiblio.org.\"><span style=\"font-weight: bold\">main site</span></a>";
}

if (is_readable ("$config->documentroot$mobi_images") && filesize ("$config->documentroot$mobi_images") > 1024) {
  if (!$newfilesys) {
    $array [] = "";
  }
  $array [] = "MOBI with images (experimental) <a href=\"$help#MOBI\" title=\"Explain MOBI.\">$helpicon</a>";
  $array [] = "";
  $array [] = "";
  $array [] = human_readable_size (filesize ("$config->documentroot$mobi_images"));
  $array [] = "<a href=\"$mobi_images\" title=\"Download from ibiblio.org.\"><span style=\"font-weight: bold\">main site</span></a>";
}

// plucker stuff

if ($offer_plucker) {
  if (!$newfilesys) {
    $array [] = "";
  }
  $array [] = "Plucker <a href=\"$help#Plucker\" title=\"Explain Plucker.\">$helpicon</a>";
  $array [] = "";
  $array [] = "";
  $array [] = "unknown";
  $array [] = "<a href=\"/cache/plucker/$fk_books/$fk_books\" 
title=\"Download from ibiblio.org.\"><span style=\"font-weight: bold\">main site</span></a>";

# gbn: mobile ebooks.  If Plucker conversion works, this should work, too:
  if (!$newfilesys) {
    $array [] = "";
  }
  $array [] = "Mobile eBooks <a href=\"$help#Mobile\" title=\"Explain Mobile.\">$helpicon</a>";
  $array [] = "";
  $array [] = "";
  $array [] = "unknown";
  $array [] = "<a href=\"mobile/mobile.php?fk_books=$fk_books\" 
title=\"Download from ibiblio.org.\"><span style=\"font-weight: bold\">main site</span></a>";

}

$table = new FilesTable ();
$table->summary = "Table of available file types and sizes.";
$table->toprows = $array;
$table->PrintTable ($db, "Formats Available For Download <a href=\"$help#Table:_Formats_Available_For_Download\" title=\"Explain this table.\">$helpicon</a>", "pgdbfiles");

echo ("</div>\n\n");

if ($offer_recode) {
  $recode_encoding = strtoupper ($recode_candidate->encoding);
  p ("\xC2\xB9 If you need a special character set, try our " . 
     "<a href=\"$config->world/recode?file=$recode_candidate->filename" . 
     "&from=$recode_encoding\" rel=\"nofollow\">" . 
     "online recoding service</a>.");
}

pagefooter (0);

// implements a page cache
// if this page is viewed it will write a static version
// into the etext cache directory
// MultiViews and mod_rewrite then take care to serve
// the static page to the next requester

$cachedir  = "$config->documentroot/cache/bibrec/$fk_books";
umask (0);
mkdir ($cachedir);
$cachefile = "$cachedir/$fk_books.html.utf8";

$hd = fopen ($cachefile, "w");
if ($hd) {
  fwrite ($hd, $output);
  fclose ($hd);
}

$hd = gzopen ("$cachefile.gz", "w9");
if ($hd) {
  gzwrite ($hd, $output);
  gzclose ($hd);
}

exit ();

?>