Tag: data sets

Archived Data Sets

Last week saw a flurry of messages about how to find archived data sets. This is the list of resources and links from those messages. The bulk of this list came from the Data Rescue Project (@datarescue2025.bsky.social) that was shared by Melissa Haendel. Please check the Data Rescue Project page for new updates. The Data Rescue Project now has a homepage https://www.datarescueproject.org/about-data-rescue-project/

Larger and Established Data / Website Efforts

End of Term Crawl 

  • The main coordinated effort to archive websites
  • Datasets have been more of a challenge, especially data embedded in databases.

EDGI

Public Environmental Data Project 

Harvard’s LIbrary Innovation Lab Team

ICPSR

  • Overview of ICPSR’s data rescue activities to date:
    • Downloaded ~2800 files from various sources requested by researchers; all the files ICPSR collected will soon be available via a dropbox link.
    • Examining CDC data dump from archive.org to assess what might be missing.
      • Ideally will also be a resource for those looking for data to see what is/isn’t available.
    • ICPSR staff and allies are generating metadata for each of the datasets we have so that we can make them available through an existing archive at ICPSR (DataLumos, openICPSR, or the Resource Center for Minority Data, depending on our timeline and some technical issues we’re working out)
  • ICPSR Data Lumos – They have the older version of a lot of major data, including a recent addition from the CDC.

IPUMS

  • They have data and have been working on cataloging efforts
  • Notification went out yesterday that they will share more soon.

Dryad

  • Generalist repository available to help with data publication, storage, and preservation.

Synapse

  • Generalist biology and biomedical data repository available to help with data publication, storage, and preservation.

Silencing Science Tracker

  • Joint initiative of the Sabin Center for Climate Change Law and the Climate Science Legal Defense Fund.
  • Tracks government attempts to restrict or prohibit scientific research, education or discussion, or the publication or use of scientific information.

OSF

  • Generalist repository for archiving, sharing, and storing all types of research outputs, not limited to preprints or only data.
  • OSF is available as an option for pre-prints of articles if, for some reason, they cannot be posted on official sources.
  • Many universities also have institutional repositories where research (articles, data, dissertations, etc) from that institution can be posted. They also have preservation mandates. An example is Penn’s ScholarlyCommons.

The Climate Mirror Project

  • Has NOAA data pulled during the 2017 data rescue.

Open Energy Data Initiative

  • A volunteer has pointed out that “key equity data” is missing from the Dept of Energy. Says they were able to find it on this site. Includes additional data from DOE.

Wayback Machine

Data Rescue Events

Smaller/Ad Hoc Rescue Efforts/ Data Archiving Activists

  • UCSB LSIT Data Mirroring
    • Mirrored and archived public data on locally hosted git server
    • Includes retrieved data sets from CDC, NIH, and NOAA
  • CDC Page on Internet Archive
    • A special archive created on IA of all CDC datasets publicly available as of January 28, 2025
    • uploaded by DataHoarders (we think)
  • Datasets in Dataverse
    • Data uploaded by the Climate Change and Health Research Coordinating Center (CAFE)
      • CAFE is looking for potentially non US based location to duplicate the contents of their collection
    • Includes CDC’s Social Vulnerability Index data.  
    • Most of what’s being placed here is data focusing on health and the environment.
    • DataRefuge from 2017 DataRefuge initiative can be opened for more deposits 
  • Safeguarding Research
    • Organizer is Henrik Schönemann; https://fedihum.org/@lavaeolus
    • There is a forum: https://safeguarding-research.discourse.group/ (admin = Henrik)
      • Based in EU, USA and global – got access to Update 1-2 PB (and more on the way) of storage & people willing to seed
      • Currently, we’ve got around 1TB of data backed up
        • Including >100.000 PDFs from academia.edu (“transgender”, “Queer Studies”, “intersex”, “nonbinary” etc. – see the forum for the full list)
        • 350GB web archive of CDC, including all 30.000 files from archive.cdc.gov And much more
        • “We’re working on providing a central index of archives, with metadata about who archived what, when, to be disseminated widely alongside torrent files and act as both a central point of coordination for archivers to assess what new work is needed, and a mass distribution channel.”
      • Possible contact to CERN, will update asap
  • Data Hoarder
    • A reddit community that is coordinating efforts to rescue data. 
  • Data Hoarding 
    • index of resources and archives related to data hoarding, web archival and self hosting. 
  • ArchiveTeam Warriors
    • They run a distributed crawler. Anyone can install it to help contribute.
    • US Federal Data page
    • Data is uploaded to Archive.org by volunteers
  • Data Liberation Project
    • Note: It looks like the project may have stalled in September 2024. Send info if you know more about them.
    • Run by BigLocalNews and MuckRock, which are good groups to follow.

Tools for Data Rescues

Library Guides to Data Rescues

Articles on current efforts

Articles for context

Existing Alternative Data Sources

Thanks to Brianne Dosch for suggesting the section and some of the bullets.

  • PolicyMap – offers a free tier that can be used to view basic information down to the tract-level, but more detailed data and functionality requires a subscription; available at some universities
  • FRED – They have some demographic data as well; free and open source
  • Census Reporter – is a free, open-source platform focused on making American Community Survey (ACS) data more accessible, including the recent upload of the 2022 1-Year ACS data
  • Esri – for mapping users, the GIS vendor publishes several U.S. Census Bureau data sets, including the ACS, through its ArcGIS Online Platform
  • IPUMS – Even when the government operates normally, many analysts turn to Minnesota Population Center products to access ACS, Current Population Survey microdata and Decennial Census data
  • Social Explorer – historical Census data and more; available at some universities
  • SimplyAnalytics – has internally processed American Community Surveys; available at some universities
  • American College of Obstetricians and Gynecologists – Hosting copies of immunization schedules and contraceptive use guidance from the CDC
  • https://www.ebi.ac.uk/ena/browser/home – The European Nucleotide Archive (ENA) provides a comprehensive record of the world’s nucleotide sequencing information, covering raw sequencing data, sequence assembly information and functional annotation. Mirrors SRA public data

Economic Indicators 

  • National League of Cities: Federal Grant Navigation Equity Dashboard 
    • This tool aggregated data from many sources – it seems to still be able to categorize disadvantaged communities (by environmental and economic standards), as well as other critical data denotations that are increasingly hard to access 
  • ALICE Economic Vitality Dashboard and Report (2022 w/ 2024 update)
    • This resource specifically provides data on work, housing, and community resources for households below the ALICE threshold (Asset Limited, Income Constrained, Employed). The data is provided by the U.S. Census Bureau’s Public Use Microdata Sample (PUMS, 202!) 
  • National Equity Atlas Dashboards
    • A data and policy tool that provides a detailed report card on racial and economic equity – this tool can provide a holistic Racial Equity Index snapchat of communities. The Atlas draws its data from a unique regional equity indicators database developed and maintained by two private institutions: PolicyLink and USC Equity Research Institute ERI.

Public Health 

  • County Health Rankings & Roadmaps (CHR&R)
    • A program of University of Wisconsin’s Population Health Institute, this data tool aims to highlight the symbiotic nature of health and equity by factoring in physical environment, social and economic indicators, clinical care, and health behaviors to health outcomes. 
      • They also recommend these additional health data platforms: 
  • City Health Dashboard
    • From NYU Langone Health, this platform provides 40+ measures of health and factors affecting health across five areas (Health Behaviors, Social and Economic Factors, Physical Environment, Health Outcomes, and Clinical Care) for 970+ cities across the U.S.