Aquifer and American Social History Online at DLF Spring Forum

There was a flurry of Aquifer and American Social History Online activity at DLF Forum in Minneapolis this week. The DLF Board meeting before the Forum included an open discussion about Aquifer's future. Will DLF have a continued interest in Aquifer elements that have been well received by the digital library community, such as the MODS Guidelines and asset action work? If so, what might DLF support look like? Might some of the work be picked up in other activities and initiatives such as Bamboo while other pieces transition to other organizations for ongoing maintenance? While nothing was settled, several people stated that it would be worth keeping an existing sandbox rather than starting again from scratch. We'll also be developing an open and transparent process (but lightweight!) to encourage an organization or organizations with service capacity to take over hosting for American History Online next year.

Several people twittered about asset actions during the BoF. People are beginning to see what kinds of services can be enabled if asset actions are available and were interested in asset actions as an implementation of OAI-ORE. It might be interesting to experiment with generating asset actions as a service for data providers, making them available for harvesting so they could be deployed in the local environment as well as the aggregation.

The panel on agile development sparked some comments too. People were especially enthusiastic about the Google maps mashup on the development server that will go live in a few weeks. The Technical Advisory Group that keeps us on track congratulated the team for scoping the project in a way that allowed us to develop useful and interesting services. It was a good meeting, despite the snow flurries early in the week!

Ruby on Rails Upgrade Woes

We were having mysterious seg faults with some long-running Rails tasks on our server, so I thought it might time for some software updates. Nothing too radical just Ruby 1.8.5 -> 1.8.6 and Rails 1.2.3 -> 1.2.6. In the process I discovered that a new version of RubyGems was available, so I thought it might be good to upgrade it at the same time, Gem 0.9.4 -> 1.0.1. The Ruby, Gem, and Rails upgrades went fine, but ultimately the Gem upgrade proved to be a bad idea. The upgrade seemed to have lost all my installed Gems, except for the just upgraded Rails and its dependencies. Some minor dismay, but no big deal I thought, I've got a list of the Gems we needed, so I'll end up updating all the Gems at the same time too. Everything seemed to be going OK, until I discovered that the ruby-openid Gem had a dependency on a method from the older versions, namely Kernel#require_gem was removed and replaced with just Kernel#gem. As far as I can tell this was nothing but a name change which begs the question why do this when it is almost guaranteed to break a bunch of other code. Anyway, I had to revert back to a previous version of RubyGems, 0.9.5, which seems to have fixed the problem. All in all a rather nerve wracking undertaking.

As I type this, a task which was previously throwing seg faults has been running for about 30 minutes without error, so my hope is that all this trouble has at least solved that problem.

Compiling ruby-xslt on Windows

After much head bashing I've gotten the ruby-xslt gem to compile on Windows. Roughly, here are the steps I went through:

1. Make sure libxml-ruby is working. See my previous blog post on installing it for Windows.
   
2. 'gem install ruby-xslt' this downloads the gem and attempts to compile and install. It fails, but it leaves all the files in your ruby\lib\...\gems folder. I got 0.9.5 version of ruby-xslt
   
3. Download the latest versions of libxml2, libxslt, and libiconv with all the binaries, includes, and libs for Windows. I copied the *.dll files into my windows\system32 folder.
4. Now the trial and error began. I had to tweak the Makefile to get the include and lib paths correct for my system.
5. I was using the MS Visual Studio (VS) tools NMAKE and CL. A major issue was the version of these tools. I started off with the VS 2005, but discovered that it links to a different version of the C runtime that is not compatible with the dependent DLLs, like libxml2.dll, etc. I had to install VS 6.0, and this seemed to solve the problem.
   
6. Next I needed to make some modifications to the xslt_lib.c file. It seemed to be using some non-ANSI C constructs that MS C compiler didn't like. These seemed to be C99 compliant stuff like dynamically allocated arrays, but apparently VS does not support C99.
7. Finally, got a compiled and linked version that works. Since by this time I didn't feel like messing with gem install scripts, I just manually moved all the files into their normal ruby directories.
   

• xslt_lib.so => \ruby\lib\ruby\site_ruby\1.8\i386-msvcrt\xml folder
• xslt.rb => \ruby\lib\ruby\site_ruby\1.8\xml

The changed files plus the pre-compiled DLL (xslt_lib.so) are available from the Aquifer SourceForge site.

Messing with Metadata

The Gladys Krieble Delmas Foundation has provided support for DLF Aquifer to perform an inventory and evaluation of metadata tools that have been created to standardize metadata or change metadata from one format to another. We seek a metadata librarian to work part time for six months to identify and test existing tools and to evaluate how feasible it might be to combine various tools into a metadata improvement workflow. This is a great opportunity to connect with the digital library community by working with the Aquifer Metadata Working Group. Details about the project and the position are posted on the Digital Library Federation website. Review of applications will begin March 10, 2008.

The date normalization utility developed at the California Digital Library has already been shown to be effective in improving consistency in the way dates are represented in metadata as collections are added to American Social History Online. Improving consistency of date formats allows for services like viewing search results by date, using the Simile Timeline, to behave reliably, providing a better user experience. The goal of the Delmas funded project is to identify and evaluate other tools that could be used to improve metadata and services, whenever collections are pooled.

What is the difference between DLF Aquifer and American Social History Online?

As we come to the end of the first year of the American Social History Online project within DLF Aquifer, we are thinking ahead to what it will take to sustain American Social History Online and what it will take to sustain Aquifer. While the names, DLF Aquifer and American Social History Online have often been used interchangeably, they are not really one and the same. Currently, American Social History Online is a suite of products and services for scholars, enabled through generous support from the Andrew W. Mellon Foundation, emerging from a segment of the digital library community working on interoperability that is DLF Aquifer.

Over time, it is possible to imagine American Social History Online as a "stand-alone" product. Not in the sense that it would be monolithic, rather that American Social History Online components will be seen as a places scholars go to do their work rather than as a digital library interoperability experiment. DLF Aquifer, having grown and released American Social History Online could join the next round of interoperability work in the community, building on lessons learned from American Social History Online.

What organizational models would support this kind of flow? I remain quite taken by the model outlined by Raymond E. Miles, Grant Miles and Charles C. Snow in Collaborative Entrepreneurship published in 2005 by Stanford University Press. Miles and colleagues describe a small, nimble entrepreneurial organization that spins off ideas and partially developed products to individual companies that participate in the collaboration.

After reading the book, I contacted Miles by e-mail. He was gracious enough to offer some tips about the kinds of problems companies are collaborating to solve such as working together to convince upstream vendors, which they all use to comply with standards. This is certainly a problem that applies equally to libraries. Miles also noted that, "the collaborative skills we discuss imagine a deeper level of learning than occurs within most teams and intra-firm networks, and we are enjoying the challenge of trying to reach that level." Something to think about in our community.

What organizations in the current research library landscape might serve as entrepreneurial hubs and what organizations might ultimately nurture and support the products and services the innovation incubators generate? In addition to ad hoc collaborations and consortia organized by geography or class of library, examples of organizations that do sponsor or generate new development include OCLC Research, the Digital Library Federation, the Council of Library and Information Resources, NISO and Ithaka. Other initiatives with less organizational structure and a more specific focus might include the Open Archives Initiative and the Dublin Core Metadata Initiative. Operating collaborative initiatives from a small, nimble organization that is perceived to be neutral does have advantages in getting things done, although obtaining resources for collaboration from participants can be challenging.

There is some overlap between innovation incubators and organizations with infrastructure to nurture and support products and services including OCLC, Ithaka, individual libraries that host services for their own constituencies and consortia that run services for groups. Other initiatives such as DSpace, Fedora and LOCKSS have grown their organizations along with their technologies. Upstream suppliers of software also run systems and services, most of which are developed for a range of customers with less community participation than would be available in a true collaboration.

In the book, Miles et al. make a distinction between collaboration and cooperation,

"...we suggest that collaboration differs from... cooperation in two main ways. First, cooperation is motivated by the benefits each party expects to receive from sharing ideas, information or resources. Therefore, while cooperative behavior may be enjoyable in its own right, it is primarily extrinsically motivated. Second, because cooperative behavior ultimately involves the pursuit of self-interest, it requires periodic or even continual assessment by each participant of the amount of trust and commitment of the other party. In collaborative relationships, on the other hand, each party is as committed to the other's interests as it is to its own, and this commitment reduces the need for the continual assessment of trust and its implications for how rewards will be divided." p.40

If DLF Aquifer is a collaborative interoperability innovation generator, it could keep running under the auspices of any organization committed to leveraging digital library resources through collaboration. American Social History Online might spin-off more fully, moving to a host that is set up to run services and support business models that can sustain them, a library or a service provider. The generate and spin-off process would be a small test case for Collaborative Entrepreneurship in the research library sector.

Expanding the pool, installing new faucets

Lately, my use of water metaphors to describe Aquifer progress has been judicious, but I can no longer resist. We are on the cusp of adding significant additional resources to the pool from University of Illinois at Urbana-Champaign (UIUC), Columbia University, Harvard University and the California Digital Library. At the same time, the Sakai and federated search "outlets" are under development.



Watch the American Social History Online website regularly for new content and new features. Be sure to try the enhancements to search that allow search results to be re-sorted and that enable search results to be broken down by subject, topic, place, name, genre, decade, year, collection, language or media type. This is functionality our early testers said they would like to see. Let us know what you think.



We were excited to learn that Colin Koteles, a student at the Graduate School of Library and Information Science at UIUC is doing a thesis on Existing Conformance to DLF/Aquifer MODS Implementation Guidelines. His analysis will be useful in many ways, including setting expectations for legacy collection readiness for Aquifer aggregation and areas to focus on in training development for metadata creation for shared collections.



Leveraging effort through collaboration continues to be the Aquifer MO. Collaboration among participants of course but also collaboration with complementary initiatives. The technology working group continues to monitor OAI/ORE work, looking for points of intersection and is pursuing ways to improve the way American Social History Online works with Zotero.

Much to look forward to in 2008.

New OAI Static Repository Gateway

A new OAI Static Repository Gateway is now available. To any potential Aquifer collection providers who were holding out because they did not have an OAI data provider: here is your chance to participate.

If you would like to add your static repository just issue an initiate request:

http://imlsdcc.grainger.uiuc.edu/gateway.net/oai.aspx?initiate=your_static_url_goes _here

The gateway will validate your repository and email you with some confirmation instructions. For details on how initiate works refer to the OAI Static Repository specification:

http://www.openarchives.org/OAI/2.0/guidelines-static-repository.htm#SR_initiating

Also for anyone who would like to host their own OAI Static Gateway Service the code is open source and can be downloaded from the UIUC OAI Metadata Harvesting Project SourceForge site. The code is available from the SourceForge SVN or as a ZIP file:

http://sourceforge.net/project/showfiles.php?group_id=47963&package_id=256028

New Energy, New Horizons

Aquifer core team members barely took a breather after a good showing for American Social History Online at DLF Fall Forum in Philadelphia. Look for our presentations, Realizing Benefits for Scholars and the Digital Library Community through DLF Aquifer to be linked from the web version of the conference schedule soon after Thanksgiving.

Everyone seems eager to get on to the next thing, getting the SRU components in place for the Sakai and federated search integrations, working on facets, fixing bugs, adding collections, integrating asset actions more tightly, planning the assessment activities, getting the word out. If you have collections to contribute, please let us know!

We are particularly cognizant that Chick's tenure with us ends in March 2009 so we want to be sure the architecture is well established and documented by then. Based on some discussions with Erik Hatcher at the Ruby Conference he attended pre-forum, Chick plans some re-engineering tweaks before getting back to facets--but I'll let him fill in the detail on that work.

As we continue development, we also continue to look for and cultivate relationships with other projects and organizations in an effort to leverage what we are doing for scholars and in the digital library community. Our commitment to collaboration and to generating useful services has inoculated us against "not invented here" syndrome. In addition to the collection registry integration mentioned in an earlier post, the MIT Simile Timeline and a date normalization tool from the California Digital Library have also been incorporated in the portal. This adaptive re-use mixed in with terrific energy, teamwork and focus is what moves us along.

Ending the Radio silence with American Social History Online Portal release

Much has been quiet on the blogging front as the core team for American Social History Online works feverishly to get the new portal ready to show at the upcoming DLF Forum in Philadelphia. The portal is looking good and you can check it out at www.dlfaquifer.org. Be warned that it is still under construction so you may experience some unpredictable results. Try it with Firefox and check out the Zotero integration. Take a look at the integration with the DLF Collections registry by clicking on the collection descriptions link and view the headings for each collection as a tag cloud by selecting "view headings". Our early evaluators asked for more visual approaches to searching and browsing, thus the SIMILEtimeline integration.

Kudos to the core team, Chick Markley, Kat Hagedorn, Susan Harum and Tom Habing for getting this done and to all the members of the Aquifer working groups who have spent countless hours organizing information for the team to use, testing and giving feedback and cheering the core team on through the development process.

New postings on the wiki

Everyone,

I've gone ahead and posted the updated project plan, as well as the initial draft of the architectural spec on the wiki. The project plan needs just a smidge of work, however the spec will need some additional updates, etc.

Alison