Resources.data.gov has replaced Project Open Data.
On January 14, 2019, the Foundations for Evidence-Based Policymaking Act ("Evidence Act"), which includes the OPEN Government Data Act, was signed into law. The Evidence Act requires the Office of Management and Budget, the Office of Government Information Services, and the General Services Administration to develop and maintain an online repository (Resources.data.gov) of tools, best practices, and schema standards to facilitate the adoption of open data practices across the Federal Government.
Archived content of Project Open Data can be accessed at its GitHub repository. You can report missing content or provide any additional feedback via GitHub or by emailing datagov@gsa.gov.
This section provides further background and resources to assist agencies in implementing the common core metadata schema.
In addition to template and sample files we also produce Common Core Metadata schema in JSON schema format. JSON schema describes a JSON data format. It can be used to automatically validate Data.json files and to generate forms for creating JSON.
A) Starting from a spreadsheet
Some agencies may initially build or maintain data catalogs through large spreadsheets. In order to then generate a data.json file from this, simply import the spreadsheet (in the form of a CSV file) into the Catalog Generator, translate the metadata as needed, and then generate an appropriately formatted JSON file.
B) Translating a data file from another schema
If your agency has a JSON catalog with a different metadata schema and need to convert it to the prescribed schema needed for the data.json file, one can import the file into Catalog Generator, adjust the metadata as needed and export the results again as the appropriately formatted JSON file.
C) Utilizing CKAN
CKAN is a popular, open-source data catalog that can be installed independently or run as a service. Even if an agency hasn’t begun a data catalog, adopting CKAN can provide a robust and feature-rich solution with which to begin maintaining an internal or external catalog. In addition to providing an effective means of organizing the agency’s data catalog, CKAN provides several opportunities for generating the needed data.json file. CKAN provides a convenient CSV export that can then be imported into the Catalog Generator for conversion into an appropriately formatted JSON file. Alternatively, CKAN provides a JSON API of the data catalog which can then be altered to provide an automatic update for the appropriately formatted JSON file.
D) Using Socrata
Socrata is a popular, open data catalog that is run as a service. Agencies that have not adapted a data catalog yet, can use Socrata without having to install or setup any servers or software. In addition to providing rich visualizations and data management features, Socrata natively supports data.json so any datasets managed through Socrata will be automatically exposed correctly. Socrata also supports all the extended metadata fields through it’s extensible metadata features.
E) Manual translating a data file from another system
Agencies may also manage their data catalogs through various database solutions such as Access, an asset management system, document management system, geoportal, or assorted proprietary systems. Regardless of which system an agency employs, it should include a fundamental export option which allows the catalog to be exported as a CSV file, which can then be imported into the Catalog Generator for conversation to an appropriately formatted JSON file. Depending on the solutions’ varied further functionality, some of them may also provide APIs which can then be configured to maintain the data.json file. If not, they may instead support XML and JSON exports, which can then be converted to the proper format.
F) Convert from CSV to JSON.
In order to generate appropriately formatted JSON, simply import an appropriately formatted CSV into the Catalog Generator, ensure that the metadata fields line up, and re-export the file.
| Label | POD | CKAN | DCAT | Schema.org |
|---|---|---|---|---|
| Title | title | title | dct:title | schema:name |
| Description | description | notes | dct:description | schema:description |
| Tags | keyword | tags | dcat:keyword | schema:keywords |
| Last Update | modified | n/a | dct:modified | schema:dateModified |
| Publisher | publisher | organization → title | dct:publisher | schema:publisher |
| Contact Name | contactPoint | maintainer | dcat:contactPoint | n/a |
| Contact Email | mbox | maintainer_email | foaf:mbox | n/a |
| Unique Identifier | identifier | id | dct:identifier | n/a |
| Public Access Level | accessLevel | n/a | n/a | n/a |
| Label | POD | CKAN | DCAT | Schema.org |
|---|---|---|---|---|
| Bureau Code | bureauCode | n/a | n/a | n/a |
| Program Code | programCode | n/a | n/a | n/a |
| Access Level Comment | accessLevelComment | n/a | n/a | n/a |
| Data Dictionary | dataDictionary | n/a | n/a | n/a |
| Download URL | accessURL | resources → url | dcat:downloadURL (recommended) or dcat:accessURL (deprecated) | schema:contentUrl |
| Endpoint | webService | resources → url | dcat:accessURL | n/a |
| Format | format | resources → mimetype | dcat:mediaType (recommended) or dct:format (deprecated) | schema:encodingFormat |
| License | license | license_title | dct:license | n/a |
| Spatial | spatial | n/a | dct:spatial | schema:spatial |
| Temporal | temporal | n/a | dct:temporal | schema:temporal |
| Label | POD | CKAN | DCAT | Schema.org |
|---|---|---|---|---|
| Release Date | issued | n/a | dct:issued | schema:datePublished |
| Frequency | accrualPeriodicity | n/a | dct:accrualPeriodicity | n/a |
| Language | language | n/a | dct:language | schema:inLanguage |
| Data Quality | dataQuality | n/a | n/a | n/a |
| Category | theme | groups | dcat:theme | schema:about |
| Related Documents | references | n/a | dct:references | n/a |
| Homepage URL | landingPage | n/a | dcat:landingPage | schema:url |
| System of Records | systemOfRecords | n/a | n/a | n/a |
Below is a set of mappings that will help users of other common metadata schemas map their content to the defined POD fields. The mapping is from the POD field to the XPATH(s) in the common metadata schema where the corresponding information may be found.
###FGDC CSDGM
Established by Office of Management and Budget Circular A-16, the Federal Geographic Data Committee (FGDC) promotes the coordinated development, use, sharing, and dissemination of geographic data. The Content Standard for Digital Geospatial Metadata aims to provide a common set of terminology and definitions for the documentation of digital geospatial data.
| POD Field | FGDC CSDGM |
|---|---|
| title | /metadata/idinfo/citation/citeinfo/title |
| description | /metadata/idinfo/descript/abstract |
| keyword | /metadata/idinfo/keywords/theme/themekey /metadata/idinfo/keywords/place/placekey /metadata/idinfo/keywords/temporal/tempkey |
| modified | /metadata/idinfo/citation/citeinfo/pubdate |
| publisher | /metadata/idinfo/citation/citeinfo/pubinfo/publish /metadata/distinfo/distrib/cntinfo/cntperp/cntper /metadata/distinfo/distrib/cntinfo/cntorgp/cntorg |
| contactPoint | /metadata/idinfo/ptcontac/cntinfo/cntorgp/cntper |
| mbox | /metadata/idinfo/ptcontac/cntinfo/cntemail |
| identifier | CSDGM does not include a unique identifier in the metadata itself. Common practice is to use a URL to the metadata document as unique identifier. |
| accessLevel | /metadata/idinfo/accconst /metadata/idinfo/useconst /metadata/distinfo/distliab |
| dataDictionary | /metadata/idinfo/keywords/theme/themekt /metadata/eainfo/overview/eadetcit |
| accessURL | /metadata/distinfo/stdorder/digform/digtopt/onlinopt/computer/networka/networkr |
| webService | /metadata/idinfo/citation/citeinfo/onlink /metadata/distinfo/stdorder/digform/digtopt/onlinopt/computer/networka/networkr |
| format | /metadata/distinfo/storder/digform/digtinfo/fname |
| license | /metadata/distinfo/distliab |
| spatial | /metadata/idinfo/spdom/bounding |
| temporal | /metadata/idinfo/timeperd/timeinfo (might be one date or range. If range, separate start/end with comma) |
###ISO 19115:2003
ISO 19115:2003 defines the schema required for describing geographic information and services. It provides information about the identification, the extent, the quality, the spatial and temporal schema, spatial reference, and distribution of digital geographic data.
| POD Field | ISO 19115:2003 |
|---|---|
| title | //gmd:identificationInfo/gmd:MD_DataIdentification/gmd:citation/gmd:CI_Citation/gmd:title/gco:CharacterString |
| description | //gmd:identificationInfo/gmd:MD_DataIdentification/gmd:abstract/gco:CharacterString |
| keyword | //gmd:topicCategory/gmd:MD_TopicCategoryCode //gmd:descriptiveKeywords/gmd:MD_Keywords/gmd:keyword/gco:CharacterString |
| modified | /gmd:MD_Metadata/gmd:dateStamp/gco:Date /gmd:MD_Metadata/gmd:dateStamp/gco:DateTime |
| publisher | //gmd:identificationInfo/gmd:MD_DataIdentification/gmd:pointOfContact/gmd:CI_ResponsibleParty/gmd:organisationName/gco:CharacterString[../../gmd:role/gmd:CI_RoleCode/@codeListValue=’publisher’] //gmd:identificationInfo/gmd:MD_DataIdentification/gmd:citation/gmd:CI_Citation/gmd:citedResponsibleParty/gmd:CI_ResponsibleParty/gmd:organisationName/gco:CharacterString[../../gmd:role/gmd:CI_RoleCode/@codeListValue=’publisher’] //gmd:distributionInfo/gmd:MD_Distribution/gmd:distributor/gmd:MD_Distributor/gmd:distributorContact/gmd:CI_ResponsibleParty/gmd:organisationName |
| contactPoint | //gmd:identificationInfo/gmd:MD_DataIdentification/gmd:pointOfContact/gmd:CI_ResponsibleParty/gmd:individualName/gco:CharacterString //gmd:identificationInfo/gmd:MD_DataIdentification/gmd:pointOfContact/gmd:CI_ResponsibleParty/gmd:organisationName/gco:CharacterString |
| mbox | //gmd:identificationInfo/gmd:MD_DataIdentification/gmd:pointOfContact/gmd:CI_ResponsibleParty/gmd:contactInfo/gmd:CI_Contact/gmd:address/gmd:CI_Address/gmd:electronicMailAddress/gco:CharacterString |
| identifier | //gmd:fileIdentifier/gco:CharacterString |
| accessLevel | //gmd:identificationInfo/gmd:MD_DataIdentification/gmd:resourceConstraints/gmd:MD_Constraints/gmd:useLimitation/gco:CharacterString |
| dataDictionary | //gmd:contentInfo/gmd:MD_FeatureCatalogueDescription/gmd:featureCatalogueCitation/gmd:CI_Citation/gmd:citedResponsibleParty/gmd:CI_ResponsibleParty/gmd:contactInfo/gmd:CI_Contact/gmd:onlineResource/gmd:CI_OnlineResource/gmd:linkage |
| accessURL | //gmd:distributionInfo/gmd:MD_Distribution/gmd:transferOptions/gmd:MD_DigitalTransferOptions/gmd:onLine/gmd:CI_OnlineResource/gmd:linkage/gmd:URL[../../gmd:function/gmd:CI_OnLineFunctionCode/@codeListValue=’download’] |
| webService | |
| format | //gmd:distributionInfo/gmd:MD_Distribution/gmd:distributionFormat/gmd:MD_Format/gmd:name/gco:CharacterString |
| license | |
| spatial | //gmd:identificationInfo/gmd:MD_DataIdentification/gmd:extent/gmd:EX_Extent/gmd:geographicElement/gmd:EX_GeographicBoundingBox |
| temporal | //gmd:identificationInfo/gmd:MD_DataIdentification/gmd:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml:TimePeriod/gml:begin/gml:TimeInstant/gml:timePosition /gmd:MD_Metadata/gmd:identificationInfo/gmd:MD_DataIdentification/gmd:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml32:TimePeriod/gml32:begin/gml32:TimeInstant/gml32:timePosition /gmd:MD_Metadata/gmd:identificationInfo/gmd:MD_DataIdentification/gmd:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml:TimePeriod/gml:beginPosition /gmd:MD_Metadata/gmd:identificationInfo/gmd:MD_DataIdentification/gmd:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml32:TimePeriod/gml32:beginPosition /gmd:MD_Metadata/gmd:identificationInfo/gmd:MD_DataIdentification/gmd:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml:TimePeriod/gml:end/gml:TimeInstant/gml:timePosition /gmd:MD_Metadata/gmd:identificationInfo/gmd:MD_DataIdentification/gmd:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml32:TimePeriod/gml32:end/gml32:TimeInstant/gml32:timePosition /gmd:MD_Metadata/gmd:identificationInfo/gmd:MD_DataIdentification/gmd:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml:TimePeriod/gml:endPosition /gmd:MD_Metadata/gmd:identificationInfo/gmd:MD_DataIdentification/gmd:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml32:TimePeriod/gml32:endPosition |
ISO 19119:2005 identifies and defines the architecture patterns for service interfaces used for geographic information, defines its relationship to the Open Systems Environment model, presents a geographic services taxonomy and a list of example geographic services placed in the services taxonomy. It also prescribes how to create a platform-neutral service specification, how to derive conformant platform-specific service specifications, and provides guidelines for the selection and specification of geographic services from both platform-neutral and platform-specific perspectives.
| POD Field | ISO 19119:2005 |
|---|---|
| title | /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/gmd:citation/gmd:CI_Citation/gmd:title/gco:CharacterString |
| description | /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/gmd:abstract/gco:CharacterString |
| keyword | /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/gmd:descriptiveKeywords/gmd:MD_Keywords/gmd:keyword/gco:CharacterString |
| modified | /gmd:MD_Metadata/gmd:dateStamp/gco:Date /gmd:MD_Metadata/gmd:dateStamp/gco:DateTime |
| publisher | /gmd:MD_Metadata/gmd:contact/gmd:CI_ResponsibleParty/gmd:organisationName/gco:CharacterString |
| contactPoint | /gmd:MD_Metadata/gmd:contact/gmd:CI_ResponsibleParty/gmd:individualName/gco:CharacterString |
| mbox | /gmd:MD_Metadata/gmd:contact/gmd:CI_ResponsibleParty/gmd:contactInfo/gmd:CI_Contact/gmd:address/gmd:CI_Address/gmd:electronicMailAddress/gco:CharacterString |
| identifier | /gmd:MD_Metadata/gmd:fileIdentifier/gco:CharacterString |
| accessLevel | /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/gmd:resourceConstraints/gmd:MD_Constraints/gmd:useLimitation/gco:CharacterString |
| dataDictionary | |
| accessURL | |
| webService | /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/srv:containsOperations/srv:SV_OperationMetadata/srv:connectPoint/gmd:CI_OnlineResource/gmd:linkage/gmd:URL |
| format | /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/srv:serviceType/gco:LocalName |
| license | |
| spatial | /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/srv:extent/gmd:EX_Extent/gmd:geographicElement/gmd:EX_GeographicBoundingBox |
| temporal | /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/srv:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml:TimePeriod/gml:begin/gml:TimeInstant/gml:timePosition /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/srv:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml32:TimePeriod/gml32:begin/gml32:TimeInstant/gml32:timePosition /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/srv:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml:TimePeriod/gml:beginPosition /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/srv:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml32:TimePeriod/gml32:beginPosition /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/srv:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml:TimePeriod/gml:end/gml:TimeInstant/gml:timePosition /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/srv:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml32:TimePeriod/gml32:end/gml32:TimeInstant/gml32:timePosition /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/srv:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml:TimePeriod/gml:endPosition /gmd:MD_Metadata/gmd:identificationInfo/srv:SV_ServiceIdentification/srv:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml32:TimePeriod/gml32:endPosition” |
The above XPATHs assume the following namespaces:
| Prefix | URI |
|---|---|
| gmd | http://www.isotc211.org/2005/gmd |
| gco | http://www.isotc211.org/2005/gco |
| gml | http://www.opengis.net/gml |
| gml32 | http://www.opengis.net/gml/3.2 |
| srv | http://www.isotc211.org/2005/srv |