Google Geo Challenge: Tools for Online Geographical Referencing of Scanned Old Maps

Google.org Geo Challenge Grant Proposal

Motivation


Old maps are very visually attractive for the general public but they also hold very valuable information for many scientific disciplines, as well as in the arts and humanities. Historians, archeologists, climatologists, glaciologists and other scientists are using old maps for their research. Very often, historical documents are used for tracking environmental and climate changes. Examples of typical use are tasks related to human impact on the forest and rainforest cover, reforestration, assessment of soil erosion processes, glacier length fluctuation measurements, sea-level changes, reclamation and flooding of coastal environments, hydro-electric projects and resettlements schemes, etc. The old maps are not only beautiful but also necessary for this kind of research.
 
In addition, through their unique ability to graphically illustrate change through time, old maps are a key tools for examining many subjects relating to human history, including industrial developments, infrastructure projects (railways, roads, canals, and air travel), socio-economic change, equality and development. Through a better understanding of these processes, old maps are useful for numerous purposes, including the better planning of public services and for future social and economic development. The ability to use maps for these purposes is critically underutilised in the current Internet environment.

Introduction: online geographic reference


An enormous number of old maps and other historical documents from private collections as well as collections of public libraries and archives is being digitized all over the world.
If the maps are published on the Internet then this is usually just as raster images and only very rarely are they geographically referenced to the place they are depicting. This is mainly because of the technical difficulty of the georeferencing process and because of the need to use GIS software.

This project will develop a collection of online tools to help with the referencing of existing images by utilizing Google Maps API and other online maps including official WMS services. We will provide an easy to use public service which would reduce the georeferencing into just two simple steps:
  1. Displaying the image of the map through it's public internet address (URL) - usually just copy&paste from the address bar in the browser window.
  2. Marking three or more control point pairs on the image and an existing online map.
Based on this user input it will be possible to generate the best fitting geographic reference for any publicly available image on the Internet. It will make given image displayable in Google Earth and Maps and it will also submit this image to a geographic search engine.

All the new tools will do is publish geographic metadata in the standardized OGC KML format for existing images. The image will then be linked and displayed from the original Internet address, without any modification or a copying. The metadata we will create to accompany it will reference it to the correct position on the Earth as well as add very valuable information about the picture.

Supported image formats and test map collections


Not only any standard image (like JPEG or PNG) will be supported for the georeferencing, but also high quality zoomable images which are published as tiles (like the maps from Afriterra Foundation, or Ryhiner collection). In the first step we will implement support for Zoomify tile format, which is very popular for publishing scanned maps because there is an easy to use tile generator and attractive flash viewer available for free. Zoomify format also has an open-source implementation of tiling utilities and even a high-quality open-source image server.

As a test collection we will process maps available in the Moll map collection of the Moravian Library Brno, which contains more than 12000 maps and vedute. Currently we have over 650 maps available online and plan to scan the rest of the collection in the near future. We will also closely cooperate with Afriterra Foundation, National Library of Scotland and people behind Ryhiner Collection. They will test our online tools on their maps and write a letter of report in the end of this grant.

Later on we would like to support more image formats used for scanned maps distributions, like the image server used for example in the Map Collection of Library of Congress and another US libraries or the image format of the David Rumsey Map Collection - a fraction from his collection is available in Google Earth already, but several thousands of maps are waiting for georeference and there is no online tool which would be able to do that.
Support for those formats is not planed during the time period of Google Geo Challenge grant, but the online tools are going to be developed with the possibility of later extension for those formats.

Collaborative geo-metadata editing


Georeferencing is a very time-consuming process. For valuable collections in public libraries the librarians are able to provide standard, authoritative metadata. But that's is not enough.
We would like to give the Internet user community the ability to cooperate on the creation of georeferencing for existing images. Our online georeferencing tool will keep a log of editing history and it will allow backtracking.

The owner of the map collection will have the right to select one georeference as the official one, but the community still will be allowed to add more contol points and correct or improve the reference. Users seeing this map will have an options to use either the official version of georeference or the community based version in case both will be available.
The owner of the map collection will be also allowed to block access to his collection and setup the sharing rights according his wishes, so some maps can be for example removed from the list.

Users will be able to select area for clipping so the map collar can be removed when stitching several map sheets together. We will use the specified clipping polygon also for calculation of the bounding box. 

The important aspect of the georeference is the ability to specify exact cartographic projections and coordinate system used for particular map. This way the georeference itself will be improvable in the time. Standard users can for the first approximation use global interactive internet maps like Google Maps to reference the scanned map, but later on a cartographer can improve the control points reference and correctly specify the algorithmic attributes of the map.
We will create a prototype for this functionality in the GeoChallenge grant as well, but this problem is complex and therefore not perfectly solvable in the time window of this grant...

Open Source projects hosted on Google App Engine


The modules developed for this grant which will be usable separately are going to be published under an open-source license at code.google.com. Modification of existing open-source projects (like GDAL, OpenLayers) will be submitted in the form of patches back to the main source tree of the original project.

Produced geo-reference metadata will be publicly available on the Internet to anyone, unless owner of the map collections decide to limit the access to them.
 

Mashups and high quality map overlay


We would like to develop an online warping server for existing online maps published by Zoomify and as static images. Exact cartographic raster warping based on map projections allow the generation of tiles for very attractive mashups like those produced by MapTiler. Please have a look at USGS Grand Canon map overlay.

We would like to use experiences from the development of MapTiler for switching the application into the internet service. This way the warped tiles will be automatically available for any online Zoomify maps with specified georeference (by our online tool).

Because image warping and reprojection is very processor intensive we will provide this service only to the subset of the maps.
We also need to obey copyright, so we will primarily support warping of the maps from our collection and the collections of partner libraries.

We will test also online dynamic reprojection and tile generation.
The maps will be available by the OpenGIS® Web Map Service (WMS).

Metadata for the future search !


This project creates excellent metadata for geographic search in the map collections. There are several projects implementing this kind of search and also a very nice prototype for search interface and map combination MapSynthetizer (demo video). This project creates everything that is necessary for this kind of functionality.

Main tools developed from this grant


  1. Online georeferencing tool with support for an improvable collaborative georeference for scanned maps. It will run on Google App Engine. 


  2. KML Server for dynamic generation of KML SuperOverlay Metadata for normal images, Zoomify tiles and TMS tiles. It will run on Google App Engine.

      Look at the static demo

  3. Online warping server with ability to provide tiles for mashups (TMS) and a gateway for access from GIS systems (by OGC WMS) to Zoomify images of map collections of partners who agree with the raster tile processing. It will run on our server.

      Look at the sample mashup

Detailed description of proposed workflow in the article


Pridal, P. & P. Zabicka (2008). Tiles as an approach to on-line publishing of scanned old maps, vedute and other historical documents. e-Perimetron Vol. 3 (1) - ISSN: 1790-3769. P10-21, http://www.e-perimetron.org/

Grantee

  
Moravian Library in Brno, Czech Republic
Deputy Director: Ing. Petr Zabicka
<zabak at mzk dot cz>, +420 541 646 115 

Legal name: Moravská zemská knihovna v Brně
Public library with non-profit status (official documents and informative translation).

Project manager and main programmer

Mgr. Petr Pridal <klokan at klokan dot cz> (Curriculum Vitae)
+41 79 8630 860, +420 777 554689

Specialization: Web technologies, Mapping, Python, Open-Source, Digital libraries.
Cooperation on development of open-source projects (GDAL, OpenLayers,...)

Experiences with similar development


We have proven experience in the development of web-applications related to maps, and we have developed several prototypes for the functionality we are proposing in this grant. We are working actively with open-source sotware for processing old maps, as can be seen from our project blog.

Some of our mashups are also included in the official Google Maps API Demo Gallery: Google Maps Tile Bounds and Coordinates and Opacity Control.

The development of the MapTiler application brought us new ideas and experience necessary for this proposed grant. MapTiler is an advanced open-source application for generating overlay tiles for Google Maps, Google Earth and other interactive web mapping applications. More info is on the project website:


MapTiler is a GUI for the GDAL2Tiles utility we developed. This tool was already used  by several institutions and individuals all over the world for publishing Google Earth overlays and generating tiles from existing geodata files.

Users are, for example, people from European Commission Joint Research Center, US Forrest Service, French Geological Survey (BRGM) and several commercial companies...

The National Library of Scotland is now publishing with this tool a collection of old Ordnance Survey maps at the website:


We acquired a good knowledge of map tile publishing for both Google Maps and Google Earth (KML SuperOverlay). We are also familiar with cartographic map projections, spatial reference systems and we have knowledge related to metadata processing (MARC21, OAI-PMH, ..).

Resources covered by this grant


Google App Engine: extended limit for traffic for developed projects, if necessary.

Full-time employment of the project manager for the period of the grant.

Funds for contract-based help with component development, testing, translation and with graphical design and usability improvements and reports from third-parties. Traveling costs for visiting collection partners.

Hardware: one powerful SMP rack-based server for the warping server hosting (tool#3), and for storing cache of the warped tiles.

Project maintainance after the grant


Most of the tools will be hosted on Google App Engine, which will reduce the after development mainteinance to minimum. Still we have a grant till 2011 which will cover necessary work on the tools and probably also improvements of results after this grant is finished.

Related projects


Moravian Library Project: OldMapsOnline.org

This project allowed specification of the needs from the library side and technical possibilities on the other side. The Google Geo Challenge funding would speed up development of the tools suggested by OldMapsOnline and allow to dramatically improve quality of the results thanks to the available funding.


European projects: EuropeanaDigMap.euINSPIRE initiative
OpenSource projects like OpenLayersMapServerIIPImageMapTilerGeoNetworkTimeMap

Existing projects for scanned map processing:

But there is always a limit for the processed raster size, usually you must upload your raster data to their servers, they don't share the ground control points with the community nor is it possible to specify the original map projection and coordinate system.

OpenSource projects we will use: Python, Django, GDAL, MapServer, MapTiler

Standards used for project development: OGC KML, SiteMap.xml, OGC WMS, OSGeo TMS, DublinCore, XML

Google technologies we will use: Google App Engine, Google Maps API, Google Earth API, Google Geo Search


Schedule with the key milestones


1st month milestones:
    Tool#1: User interface design & conversion into static HTML/CSS/JavaScript
    Tool#1: Client side JavaScript development (Google Maps API, OpenLayers)
    Tool#3: Basic GDAL driver for remote Zoomify tiles is developed and tested

2nd month milestones:
    Tool#1: Zoomify layer functionality accepted by OpenLayers.org project
    Tool#1: Data model for map georeferencing is designed
    Tool#2: Prototype of dynamic KML generation for standard images (affine transformation)
    Tool#3: Server is installed in MZK, including MapServer, GDAL + new driver 

3rd month milestones:
    Tool#1: Core functionality of the georeferencing tool is implemented, data model finished.
    Tool#1: Functionality: opening by URL, control point specification, calculation of the affine transformation, export to World file, correct link to KML server (Tool#2)
    Tool#2: KML server able to dynamically generate SuperOverlay for Zoomify tiles with affine transformation from GAE
    Tool#3: Warping server provides slow dynamic WMS from remote Zoomify tiles
   
Mid-term evaluation: all three tools have functional prototypes, testers are able to submit maps, specify control points and see results of georeference in Google Earth

4th month milestones:
    Tool#1: Collection administrator functionality (copyrights, restrictions of the access to maps)
    Tool#1: Backtracking for control points possible
    Tool#1: Exports: Metadata, KML & Sitemap.xml, GeoRSS for modifications
    Tool#2: Finalized Zoomify, deployed at GAE.
    Tool#3: Installation and tests of TileCache for Zoomify trough WMS, caching in GDAL, batch warping tested

5th month milestones:
    Tool#1: Interface for the Google Geo Search based on KML exports 
    Tool#2: TMS tiles supported, tweaking of the SuperOverlay (maxLodPixels,..)
    Tool#3: Cached WMS, Batch rendering as alternative

6th month milestones:
    Tool#1: User help and reports from the partners and testers.
    Tool#2: Documentation (code.google.com wiki)
    Tool#3: Final report with speed results. 
    Screencast with demonstration of the tools. Presentations.

Final evaluation
Ċ
Petr Pridal,
Dec 20, 2008, 7:21 AM
Ċ
Petr Pridal,
Dec 18, 2008, 4:24 AM
Ċ
Petr Pridal,
Dec 18, 2008, 4:25 AM
Ċ
Petr Pridal,
Dec 18, 2008, 4:31 AM
Ċ
Petr Pridal,
Dec 20, 2008, 1:59 PM
Comments