wiki:NYSRFPC122807

Background

This is a public comment that was originally submitted in the lead up to the May 2008 Electronic Documents Report to the NY State Governor and Legislature.

In 2007 and early 2008, New York State issued a Request For Public Comment concerning electronic record policy for government documents. This is an attempt to craft a response to that call for comments. The questions asked by the State appear here as numbered, blockquoted text, like this:

  1. Example question

The public comment period ran until 2008-01-18. If you had comments for the state to read, you were expected to mail them to erecords-study@oft.state.ny.us. You should also publish them so others can read the arguments.

If you feel that these arguments resonate with you, feel free to copy or cite them in your responses to the State. And of course, this is the web: you should always feel free to link to this page.

Freedom as a Fundamental Goal

I believe that New York State (NYS) should explicitly require the use of free and open standards for all electronic/digital records, and these standards should apply to all communications entered into by the state, including:

When I say free, I should be clear that I'm referring to liberty, not price. The standards and tools promoted/required by the public sector must not be held under the control of a private organization. Anyone should be able to make use of the technology in question, including modification and redistribution of any tools required to access the data.

There should be no proprietary hurdles to jump in order to have access to government data or records. This should be an explicit state mandate.

Below, I detail some of the specifics about why choosing free standards is important.

Encouraging public access

  1. What mechanisms and processes should the State of New York

establish for accessing and reading its electronic records in order to encourage public access to those records?

Requiring free and open mechanisms for accessing public data will encourage public access because there are no proprietary requirements to be met in order to view the data.

For example, no NYS World Wide Web site should ever require the use of Microsoft Internet Explorer. This would cause problems for citizens who have not purchased a Microsoft operating system, since IE only runs on Microsoft's Windows (IE has not been supported on Mac OS for years, and has never been available on GNU/Linux or any other operating system). Instead, NYS websites should adhere to the W3C's XHTML 1.0 standard. Users of all modern computer systems have access to free tools which can render this public standard, so no one is disenfranchised.

A more subtle example: no NYS electronic records should be stored or published in Adobe's Flash format (a.k.a. swf). While Adobe makes players for this format available without charge for most modern operating systems, users are not allowed to modify or redistribute these players, and the license of the format specification itself prohibits use of the specification to make another player. This lack of liberty (despite the absence of immediate financial cost for those platforms supported by Adobe) means that the public is required to make some sort of arrangement with Adobe (a specific private company) in order to access public data. Since Adobe has no requirement to interact with the public in an evenhanded way, and citizens have no legal recourse to accessing the data themselves, they are effectively discouraged from accessing it.

Encouraging interoperability

  1. What mechanisms and processes should the State of New York

establish for accessing and reading its electronic records to encourage interoperability and data sharing with citizens, business partners and other jurisdictions?

Free formats, standards, and tools encourage interoperability and data sharing because there is no restriction on adoption for the other parties in communication.

If NYS was to choose a proprietary format for electronic records, it would need to pay the proprietor of that format a fee for its use. If a neighboring jurisdiction was to choose a proprietary format for electronic records, it would also need to pay the proprietor a fee for its use. If the two jurisdictions happened to choose different proprietary formats, then both jurisdictions will need to pay fees to both vendors if interoperability is desired.

While this would be a windfall for the vendors, the sum of the costs to all jurisdictions scales exponentially as the number of jurisdictions desiring mutual interoperability grows. Better to choose free interchange standards so that there are no additional per-jurisdiction per-vendor costs due to the proprietary nature of the records.

Encouraging appropriate government control

  1. What mechanisms and processes should the State of New York

implement to encourage appropriate government control of its electronic records?

Free formats, standards, and tools allow governments (and other entities) to retain control over their own data.

As a fictional example, consider sanitation district boundary records stored in GBG format in 1995. GBG was (fictionally) owned at the time by Dispatch Service Co. (DSCo), which specialized in sanitation logistics. Fast forward to 2008: DSCo has decided that it does not want to support the format any longer (or has been sold to a parent company uninterested in sanitation logistics, or has simply collapsed). The State is now in a weak position to have any access to tools needed for modifying garbage truck routes.

Choosing free formats and free tools from the beginning would allow the government better control over the data because it would be insulated from the fate of any particular instance of DSCo.

Encouraging choice and vendor neutrality

  1. What mechanisms and processes should the State of New York

consider for encouraging choice and vendor neutrality when creating, maintaining, exchanging and preserving its electronic records?

Free formats, standards, and tools avoid vendor-lock-in and promote competitive industry.

As in the above example with sanitation logistics, choosing free formats and free tools would let NYS negotiate among any vendor to meet its sanitation dispatch needs. Even if DSCo was still a thriving, healthy corporation, if the formats and tools used were all free, it would be forced to compete with any other entity willing and able to do the work needed by the city. The underlying freedom translates into a better negotiating position for the State, and encourages healthy competition and interoperability among the potential vendors.

Electronic record life cycle

  1. Are there mechanisms and processes the State of New York should

establish that are specific to the management of its electronic records in its various life cycle stages (creation, maintenance, exchange, preservation and disposal)?

Free formats, standards, and tools preserve the integrity of electronic records, and enable verifiable disposal.

When the underlying technologies are unencumbered by any restrictive vendor-applied rules, it becomes much easier to keep data up-to-date, and to be certain that destroyed data is actually destroyed.

If you want to ensure that records can be kept up-to-date, again choosing free formats, standards, and tools helps you to make changes and negotiate between vendors.

Similarly with record disposal. Only unfettered, transparent access to your systems can verify that no trace of the record remains. While i don't expect complete destruction of records to be a high priority for most government tasks, it is extremely important in some cases. For example, court-ordered destruction of data gathered by unconstitutional methods should be verifiable. With a proprietary toolset, it is significantly more difficult to verify that a record has been completely purged simply because no one but the vendor of that tool knows exactly what it does.

Data preservation

  1. How should the State address the long term preservation of its

electronic records? What should the State consider regarding public access to such archived content?

Free formats, standards, and tools make it easier to transfer any electronic records to new media, and ensure that the government and the public of the future will be able to interpret the data. Can you read a Lotus 123 spreadsheet (or any other obscure-today, once-popular proprietary format) on your computer today? Can you read an old ASCII text file? The free, well-documented, legally-unencumbered ASCII format is much more accessible now, and will continue to be in the future. Similarly, choosing UTF-8 and the Open Document Format today will give a better shot for the readers of tomorrow.

Preserving data for your own future consumption is very similar to sending data to a relatively-sophisticated party with whom you only have one-way communications. Remember that you don't have any way of knowing if the particular operating system, interface, or libraries are available to your future self. If you want that remote party to be able to understand your data, you are better off using well-documented, freely-available tools and formats. Your future self will have access to these because they are already widely documented and legally unencumbered.

Specific statute changes

  1. What changes, if any, should be made to the government records

management provisions in New York Statutes? (Please reference those laws which are cited here: http://www.archives.nysed.gov/a/records/mr_laws.shtml).

I wish I had a specific suggestion for modification of the Statutes, but my experience and skill lies in working with electronic data, not in working with laws or legal language. So at the moment, I'll refrain from making specific suggestions for legal language.

Implementation costs

  1. What constraints and benefits should the State of New York

consider regarding the costs of implementing a comprehensive plan for managing its electronic records?

There will be significant costs to State-wide adoption of any electronic record plan, whether proprietary or free. Some proprietary vendors will most likely try to offer short-term cost discounts to encourage the State to choose a format or a system that they control. Accepting such an offer would be a very bad deal for the State and for the citizenry in the long run, due to the problems with proprietary tools, formats, and standards outlined here.

Transitioning to free infrastructure should have similar real costs to transitioning to any particular proprietary infrastructure (less of course any surcharge that the proprietors levy on the latter). But choosing freedom will have many more payoffs in the future (see question 11).

Highly-specialized data formats

  1. What should the State of New York consider regarding the

management of highly specialized data formats such as CAD, digital imaging, Geographic Information Systems and multimedia?

Choosing free data formats wherever possible will ensure the widest range of tools to be able to work with any given dataset, even in highly-specialized domains.

Digital imaging and multimedia have significant, functional free options in the wild. For example, images can be stored and transmitted at arbitrarily high resolution in the Portable Network Graphics format. Video and audio can be stored and transmitted using the unencumbered formats provided by the Xiph foundation. GIS data can use a variety of free formats, including the widely-available TIGER format. XML-based free formats like SVG can also be used for arbitrarily-detailed vector graphics.

Where there is no free format, standard, or tool available (this would be a a highly unusual case), the State should collaborate with jurisdictions with similar needs to craft a free format to be published for public review. In this way, NYS can retain the benefits of free infrastructure for its own data, and ensure that its own needs are met in the future without being locked into a single vendor.

Ongoing costs and savings

  1. What constraints and benefits should the State of New York

consider regarding potential savings or additional costs associated with the management of defined electronic record formats?

Choosing free formats, standards, and tools will provide significant cost savings as time goes on and State-wide policy stabilizes.

Because of the lack of vendor lock-in, the State will be able to choose from a healthy range of groups offering services to maintain and manipulate the data. No overhead or maintenance will need to be paid to any particular vendor. And as new needs arise, the State can employ teams of their choice to craft new tools or modify old ones without needing the permission of any particular proprietor.

Additionally, by making a public commitment to free tools the State avoids the risk of becoming embroiled in litigation concerning breach of licensing. Even threats from litigious proprietary vendors for breach of license can prove costly, as the city of Philadelphia found in 2001.

Existing policies as precedent

  1. What existing policies and procedures in the private or public

sector for the management of electronic records would be appropriate for the State of New York to examine? Please cite specific examples.

Massachusetts recently adopted the OpenDocument format as its standard document format. While their decision to retain Microsoft Office and use an ODF plugin was unfortunate, their selection of a free format allows them the ability to move to free software in the future. Unfortunately, their more-recent decision to allow the use of Microsoft's OOXML format (a non-free format because of its numerous hidden "gotchas" and its control by a single vendor, despite Microsoft's claim to the contrary) is deplorable, and returns them to the very real possibility of one-vendor lock-in.

Minnesota's house bill H3971 from Legislative session 84 (2005-2006) was a concrete proposal for a State mandate for free document formats, but I don't think it was passed. It would be good to see New York take a similar position to the one proposed by Rep. Thissen.

Brazil (english translation by google) and Croatia have also in recent years made explicit government mandates for free standards and free software. While the implementations have not been as vigorous as they could be, but following their intent in NYS would be admirable.

The EU has also issued the European Interoperability Framework for pan-European eGovernment Services, which states principles to be adopted by governmental services, including:

  • Use of Open Standards
  • Assess the benefits of Open Source Software
  • Use of Multilateral Solutions

While this framework offers little in the way of concrete guidance, their general principles seem sound.

The Extremadura region in Spain adopted government-wide free software policies in 2006.

Mannheim, Germany began transitioning their computing infrastructure to Linux in 2006.

The City Council of Munich mandated a switch to Free and Open Source software in 2003 This transition project is now under way, known as LiMux.

Adequacy of existing policy

  1. Are New York State's existing standards, regulations and

guidelines regarding records management adequate to meet the challenges of electronic records retention? How should these standards, regulations and guidelines be changed?

As with question number 8, I'm going to decline to offer specific commentary on general legal language since it is outside my particular expertise. I expect to see a strong mandate for freedom in any newly-created modified standards, regulations, and guidelines. I hope that New York State legal and legislative representatives will keep these goals in mind while drafting new requirements.

Last modified 9 years ago Last modified on May 21, 2008, 11:28:54 PM