{
  "_schema": "https://data.nist.gov/od/dm/nerdm-schema/v0.7#",
  "@context": [
    "https://data.nist.gov/od/dm/nerdm-pub-context.jsonld",
    {
      "@base": "ark:/88434/mds2-3172"
    }
  ],
  "@type": [
    "nrdp:DataPublication",
    "nrdp:PublicDataResource",
    "dcat:Dataset"
  ],
  "_extensionSchemas": [
    "https://data.nist.gov/od/dm/nerdm-schema/pub/v0.7#/definitions/PublicDataResource"
  ],
  "@id": "ark:/88434/mds2-3172",
  "ediid": "ark:/88434/mds2-3172",
  "version": "1.0.1",
  "doi": "doi:10.18434/mds2-3172",
  "title": "JARVIS-Leaderboard: Large Scale Benchmark of Materials Design Methods",
  "contactPoint": {
    "fn": "Daniel Wines",
    "hasEmail": "mailto:daniel.wines@nist.gov"
  },
  "modified": "2024-02-07",
  "status": "available",
  "landingPage": "https://data.nist.gov/od/id/mds2-3172",
  "description": [
    "Lack of rigorous reproducibility and validation are major hurdles for scientific development across many fields. Materials science in particular encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues. However, a comprehensive comparison and benchmarking on an integrated platform with multiple data modalities with both perfect and defect materials data is still lacking. This work introduces JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC) and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the inter-laboratory approach to establish benchmarks. There are 1281 contributions to 274 benchmarks using 152 methods with more than 8 million data-points, and the leaderboard is continuously expanding."
  ],
  "keyword": [
    "benchamrks",
    "materials science"
  ],
  "theme": [
    "Chemistry: Theoretical chemistry and modeling",
    "Materials: Modeling and computational material science"
  ],
  "topic": [
    {
      "@type": "Concept",
      "scheme": "https://data.nist.gov/od/dm/nist-themes/v1.1",
      "tag": "Chemistry: Theoretical chemistry and modeling"
    },
    {
      "@type": "Concept",
      "scheme": "https://data.nist.gov/od/dm/nist-themes/v1.1",
      "tag": "Materials: Modeling and computational material science"
    },
    {
      "scheme": "https://data.nist.gov/od/dm/nist-themes-chipsmetrology/v2.0",
      "tag": "Metrology: Computational Technique: Simulation",
      "@type": "Concept"
    },
    {
      "scheme": "https://data.nist.gov/od/dm/nist-themes-chipsmetrology/v2.0",
      "tag": "Materials Research: Semiconductor",
      "@type": "Concept"
    },
    {
      "scheme": "https://data.nist.gov/od/dm/nist-themes-chipsmetrology/v2.0",
      "tag": "Materials Research: Dielectric and Insulator",
      "@type": "Concept"
    },
    {
      "scheme": "https://data.nist.gov/od/dm/nist-themes-chipsmetrology/v2.0",
      "tag": "Materials Research: Metal",
      "@type": "Concept"
    },
    {
      "scheme": "https://data.nist.gov/od/dm/nist-themes-chipsmetrology/v2.0",
      "tag": "Metrology: Grand Challenge: 4-Modeling and Simulating Semiconductor Materials, Designs, and Components",
      "@type": "Concept"
    }
  ],
  "references": [
    {
      "@type": [
        "npg:Document"
      ],
      "@id": "#ref:10.1038/s41524-024-01259-w",
      "refType": "IsSupplementTo",
      "location": "https://doi.org/10.1038/s41524-024-01259-w",
      "_extensionSchemas": [
        "https://data.nist.gov/od/dm/nerdm-schema/bib/v0.7#/definitions/DCiteReference"
      ],
      "title": "JARVIS-Leaderboard: a large scale benchmark of materials design methods",
      "issued": "2024-05-07",
      "citation": "Choudhary, K., Wines, D., Li, K., Garrity, K. F., Gupta, V., Romero, A. H., Krogel, J. T., Saritas, K., Fuhr, A., Ganesh, P., Kent, P. R. C., Yan, K., Lin, Y., Ji, S., Blaiszik, B., Reiser, P., Friederich, P., Agrawal, A., Tiwary, P., \u00e2\u0080\u00a6 Tavazza, F. (2024). JARVIS-Leaderboard: a large scale benchmark of materials design methods. Npj Computational Materials, 10(1). https://doi.org/10.1038/s41524-024-01259-w\n"
    }
  ],
  "accessLevel": "public",
  "license": "https://www.nist.gov/open/license",
  "publisher": {
    "name": "National Institute of Standards and Technology",
    "@type": "org:Organization"
  },
  "language": [
    "en"
  ],
  "bureauCode": [
    "006:55"
  ],
  "programCode": [
    "006:052"
  ],
  "components": [
    {
      "accessURL": "https://github.com/usnistgov/jarvis_leaderboard",
      "title": "JARVIS-Leaderboard",
      "@type": [
        "nrdp:AccessPage",
        "dcat:Distribution"
      ],
      "@id": "#usnistgov/jarvis_leaderboard",
      "_extensionSchemas": [
        "https://data.nist.gov/od/dm/nerdm-schema/pub/v0.7#/definitions/AccessPage"
      ]
    },
    {
      "@id": "cmps/jarvis_leaderboard-main.zip",
      "@type": [
        "nrdp:DataFile",
        "nrdp:DownloadableFile",
        "dcat:Distribution"
      ],
      "_extensionSchemas": [
        "https://data.nist.gov/od/dm/nerdm-schema/pub/v0.7#/definitions/DataFile"
      ],
      "filepath": "jarvis_leaderboard-main.zip",
      "downloadURL": "https://data.nist.gov/od/ds/mds2-3172/jarvis_leaderboard-main.zip",
      "mediaType": "application/x-zip-compressed",
      "title": "Large Scale Benchmark of Materials Design Methods",
      "size": 236786213,
      "checksum": {
        "hash": "453c68b6a131c322cb3d816e67a9dabc7ed4e0bbd463e34949fd1eb4ae2876e5",
        "algorithm": {
          "tag": "sha256",
          "@type": "Thing"
        }
      }
    }
  ],
  "_editStatus": "done",
  "authors": [
    {
      "familyName": "Choudhary",
      "fn": "Kamal  Choudhary",
      "givenName": "Kamal",
      "middleName": "",
      "affiliation": [
        {
          "title": "Johns Hopkins University",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0001-9737-8074",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Wines",
      "fn": "Daniel  Wines",
      "givenName": "Daniel",
      "middleName": "",
      "affiliation": [
        {
          "title": "National Institute of Standards and Technology",
          "@type": "org:Organization",
          "@id": "ror:05xpvk416"
        }
      ],
      "orcid": "0000-0003-3855-3754",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Li",
      "fn": "Kangming   Li",
      "givenName": "Kangming ",
      "middleName": "",
      "affiliation": [
        {
          "title": "University of Toronto",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0003-4471-8527",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Garrity",
      "fn": "Kevin F. Garrity",
      "givenName": "Kevin",
      "middleName": "F.",
      "affiliation": [
        {
          "title": "National Institute of Standards and Technology",
          "@type": "org:Organization",
          "@id": "ror:05xpvk416"
        }
      ],
      "orcid": "0000-0003-0263-4157",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Gupta",
      "fn": "Vishu  Gupta",
      "givenName": "Vishu",
      "middleName": "",
      "affiliation": [
        {
          "title": "Northwestern University",
          "@type": "org:Organization"
        },
        {
          "title": "Princeton University",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-4931-7194",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Romero",
      "fn": "Aldo H. Romero",
      "givenName": "Aldo",
      "middleName": "H.",
      "affiliation": [
        {
          "title": "West Virginia University",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0001-5968-0571",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Krogel",
      "fn": "Jaron T. Krogel",
      "givenName": "Jaron",
      "middleName": "T.",
      "affiliation": [
        {
          "title": "Oak Ridge National Laboratory ",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-1859-181X",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Saritas",
      "fn": "Kayahan  Saritas",
      "givenName": "Kayahan",
      "middleName": "",
      "affiliation": [
        {
          "title": "Oak Ridge National Laboratory ",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-2240-8520",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Fuhr",
      "fn": "Addis  Fuhr",
      "givenName": "Addis",
      "middleName": "",
      "affiliation": [
        {
          "title": "Oak Ridge National Laboratory ",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-8819-8255",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Ganesh",
      "fn": "Panchapakesan  Ganesh",
      "givenName": "Panchapakesan",
      "middleName": "",
      "affiliation": [
        {
          "title": "Oak Ridge National Laboratory ",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-7170-2902",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Kent",
      "fn": "Paul R.C. Kent",
      "givenName": "Paul",
      "middleName": "R.C.",
      "affiliation": [
        {
          "title": "Oak Ridge National Laboratory ",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0001-5539-4017",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Yang",
      "fn": "Keqiang  Yang",
      "givenName": "Keqiang",
      "middleName": "",
      "affiliation": [
        {
          "title": "Texas A&M University",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0009-0009-9286-9259",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Lin",
      "fn": "Yuchao  Lin",
      "givenName": "Yuchao",
      "middleName": "",
      "affiliation": [
        {
          "title": "Texas A&M University",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0009-0003-6271-6397",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Ji",
      "fn": "Shuiwang  Ji",
      "givenName": "Shuiwang",
      "middleName": "",
      "affiliation": [
        {
          "title": "Texas A&M University",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-4205-4563",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Blaiszik",
      "fn": "Ben  Blaiszik",
      "givenName": "Ben",
      "middleName": "",
      "affiliation": [
        {
          "title": "University of Chicago",
          "@type": "org:Organization"
        },
        {
          "title": "Argonne National Laboratory",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-5326-4902",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Reiser",
      "fn": "Patrick  Reiser",
      "givenName": "Patrick",
      "middleName": "",
      "affiliation": [
        {
          "title": "Karlsruhe Institute of Technology",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-7052-696X",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Friederich",
      "fn": "Pascal  Friederich",
      "givenName": "Pascal",
      "middleName": "",
      "affiliation": [
        {
          "title": "Karlsruhe Institute of Technology",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0003-4465-1465",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Agrawal",
      "fn": "Ankit  Agrawal",
      "givenName": "Ankit",
      "middleName": "",
      "affiliation": [
        {
          "title": "Northwestern University",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-5519-0302",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Tiwary",
      "fn": "Pratyush   Tiwary",
      "givenName": "Pratyush ",
      "middleName": "",
      "affiliation": [
        {
          "title": "University of Maryland, College Park",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-2412-6922",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Beyerle",
      "fn": "Eric  Beyerle",
      "givenName": "Eric",
      "middleName": "",
      "affiliation": [
        {
          "title": "University of Maryland, College Park",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0001-5717-3197",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Minch",
      "fn": "Peter  Minch",
      "givenName": "Peter",
      "middleName": "",
      "affiliation": [
        {
          "title": "Rensselaer Polytechnic Institute",
          "@type": "org:Organization"
        }
      ],
      "orcid": "",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Rhone",
      "fn": "Trevor  David Rhone",
      "givenName": "Trevor ",
      "middleName": "David",
      "affiliation": [
        {
          "title": "Rensselaer Polytechnic Institute",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-0198-9952",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Takeuchi",
      "fn": "Ichiro   Takeuchi",
      "givenName": "Ichiro ",
      "middleName": "",
      "affiliation": [
        {
          "title": "University of Maryland, College Park",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0003-2625-0553",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Wexler",
      "fn": "Robert  B. Wexler",
      "givenName": "Robert ",
      "middleName": "B.",
      "affiliation": [
        {
          "title": "Washington University in St. Louis",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-6861-6421",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Mannodi-Kanakkithodi",
      "fn": "Arun   Mannodi-Kanakkithodi",
      "givenName": "Arun ",
      "middleName": "",
      "affiliation": [
        {
          "title": "Purdue University",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0003-0780-1583",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Ertekin",
      "fn": "Elif  Ertekin",
      "givenName": "Elif",
      "middleName": "",
      "affiliation": [
        {
          "title": "University of Illinois Urbana-Champaign",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-7816-1803",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Mishra",
      "fn": "Avanish   Mishra",
      "givenName": "Avanish ",
      "middleName": "",
      "affiliation": [
        {
          "title": "Los Alamos National Laboratory",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0003-3997-0445",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Mathew",
      "fn": "Nithin   Mathew",
      "givenName": "Nithin ",
      "middleName": "",
      "affiliation": [
        {
          "title": "Los Alamos National Laboratory",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-2316-3190",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Wood",
      "fn": "Mitchell   Wood",
      "givenName": "Mitchell ",
      "middleName": "",
      "affiliation": [
        {
          "title": "Sandia National Laboratories",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0001-5878-4096",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Rohskopf",
      "fn": "Andrew   Rohskopf",
      "givenName": "Andrew ",
      "middleName": "",
      "affiliation": [
        {
          "title": "Sandia National Laboratories",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0002-2712-8296",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Hattrick-Simpers",
      "fn": "Jason   Hattrick-Simpers",
      "givenName": "Jason ",
      "middleName": "",
      "affiliation": [
        {
          "title": "University of Toronto",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0003-2937-3188",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Wang",
      "fn": "Shih-Han   Wang",
      "givenName": "Shih-Han ",
      "middleName": "",
      "affiliation": [
        {
          "title": "Virginia Polytechnic Institute and State University",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0003-4418-2080",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Achenie",
      "fn": "Luke  E. K.  Achenie",
      "givenName": "Luke ",
      "middleName": "E. K. ",
      "affiliation": [
        {
          "title": "Virginia Polytechnic Institute and State University",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0001-9850-5346",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Xin",
      "fn": "Hongliang   Xin",
      "givenName": "Hongliang ",
      "middleName": "",
      "affiliation": [
        {
          "title": "Virginia Polytechnic Institute and State University",
          "@type": "org:Organization"
        }
      ],
      "orcid": "0000-0001-9344-1697",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Williams",
      "fn": "Maureen   Williams",
      "givenName": "Maureen ",
      "middleName": "",
      "affiliation": [
        {
          "title": "National Institute of Standards and Technology",
          "@type": "org:Organization",
          "@id": "ror:05xpvk416"
        }
      ],
      "orcid": "0000-0001-9144-0551",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Biacchi",
      "fn": "Adam  J. Biacchi",
      "givenName": "Adam ",
      "middleName": "J.",
      "affiliation": [
        {
          "title": "National Institute of Standards and Technology",
          "@type": "org:Organization",
          "@id": "ror:05xpvk416"
        }
      ],
      "orcid": "0000-0001-5663-2048",
      "@type": "foaf:Person"
    },
    {
      "familyName": "Tavazza",
      "fn": "Francesca   Tavazza",
      "givenName": "Francesca ",
      "middleName": "",
      "affiliation": [
        {
          "title": "National Institute of Standards and Technology",
          "@type": "org:Organization",
          "@id": "ror:05xpvk416"
        }
      ],
      "orcid": "0000-0002-5602-180X",
      "@type": "foaf:Person"
    }
  ],
  "annotated": "2025-11-04T18:02:04.815289",
  "revised": "2025-09-24T15:26:57.200120",
  "issued": null,
  "firstIssued": "2025-09-24T15:26:57.200120",
  "releaseHistory": {
    "@id": "ark:/88434/mds2-3172/pdr:v",
    "@type": [
      "nrdr:ReleaseHistory"
    ],
    "hasRelease": [
      {
        "version": "1.0.0",
        "issued": "2025-09-24",
        "@id": "ark:/88434/mds2-3172/pdr:v/1.0.0",
        "location": "https://data.nist.gov/od/id/ark:/88434/mds2-3172/pdr:v/1.0.0",
        "description": "initial release"
      },
      {
        "version": "1.0.1",
        "issued": "2025-11-04",
        "@id": "ark:/88434/mds2-3172/pdr:v/1.0.1",
        "location": "https://data.nist.gov/od/id/ark:/88434/mds2-3172/pdr:v/1.0.1",
        "description": "added to CHIPS Metrology collection"
      }
    ]
  },
  "isPartOf": [
    {
      "@type": [
        "nrda:ScienceTheme",
        "nrdp:PublicDataResource"
      ],
      "@id": "ark:/88434/pdr0-0002",
      "title": "CHIPS Metrology Data"
    }
  ]
}