C2M2

Table of Contents

• Introduction
• Resources
• Frictionless Data Packages
• Identifiers
• C2M2 Tables
• Reference Tables
• Submission Prep Script
• Datapackage Submission
• Tutorial

Introduction

Schematic of C2M2

(please right-click on the image to see options to view a larger image in a different tab)

The Crosscut Metadata Model (C2M2) is a flexible metadata standard for describing experimental resources in biomedicine and related fields. A complete C2M2 submission, also called an "instance" or a "datapackage", is a zipped folder containing multiple tab-delimited files (TSVs) representing metadata records along with a JSON schema. To read more about datapackages, skip to the Frictionless Data Packages section.

Each TSV file is a data table containing various data records (rows) and their values for different fields (columns). Entity tables describe various types of data objects, while association tables describe the relationships between different entities.

This page is adapted from the original C2M2 documentation developed by the CFDE Coordination Center (CFDE-CC).

Resources

• The c2m2-frictionless-dataclass
  • This repository includes the c2m2-frictionless Python package, which contains specific helper functions for C2M2 datapackage building and validation.
  • This package is not designed to generate a complete C2M2 datapackage from any given data, but should be used in collaboration with the provided schema and ontology preparation scripts, as well as with DCC-specific scripts.
• The most up-to-date C2M2 JSON schema
• The most up-to-date C2M2 ontology preparation script and files
• The original C2M2 documentation from the CFDE Coordination Center contains more details on the concepts discussed here.

Frictionless Data Packages

C2M2 instances are also known as "datapackages" based on the Data Package meta-specification from Frictionless Data.

From the original C2M2 documentation:

The Data Package meta-specification is a platform-agnostic toolkit for defining format and content requirements for files so that automatic validation can be performed on those files, just as a database management system stores definitions for database tables and automatically validates incoming data based on those definitions. Using this toolkit, the C2M2 JSON Schema specification defines foreign-key relationships between metadata fields (TSV columns), rules governing missing data, required content types and formats for particular fields, and other similar database management constraints. These architectural rules help to guarantee the internal structural integrity of each C2M2 submission, while also serving as a baseline standard to create compatibility across multiple submissions received from different DCCs.

Identifiers

In order to standardize and integrate information across DCCs, there must be a system of assigning unambiguous identifiers to individual DCC concepts and resources. These are the "C2M2 IDs", consisting of a id_namespace prefix and local_id suffix. Additionally, the C2M2 also allows individual resources to be assigned a persistent_id.

From the original C2M2 documentation:

Optional persistent_id identifiers are meant to be stable enough to be scientifically cited, and to provide for further investigation by accessing related resolver services. To be used as a C2M2 persistent_id, an ID

1. will represent an explicit commitment by the managing DCC that the attachment of the ID to the resource it represents is permanent and final
2. must be a format-compliant URI or a compact identifier, where the protocol (the "scheme" or "prefix") specified in the ID is registered with at least one of the following (see the given lists for examples of URIs and compact identifiers)
   • the IANA (list of registered schemes)
   • scheme used must be assigned either "Permanent" or "Provisional" status
   • Identifiers.org (list of registered prefixes)
   • N2T (Name-To-Thing) (list of registered prefixes)
   • protocols not appearing in the above registries but explicitly approved by the CFDE-CC. Currently, this list is limited to one protocol, namely drs:// URIs identifying GA4GH Data Repository Service resources.
3. if representing a file, an ID used as a persistent_id cannot be a direct-download URL for that file: it must instead be an identifier permanently attached to the file and only indirectly resolvable (through the scheme or prefix specified within the ID) to the file itself

C2M2 Tables

Sourced from the CFDE-CC Documentation Wiki

All of the tables in a C2M2 datapackage are inter-linked via foreign key relationships, as shown in the following diagram of the complete C2M2 system.

Entity relationship diagram of C2M2

(please right-click on the image to see options to view a larger image in a different tab)

Reference Tables

Certain table fields within the C2M2 are restricted to only a few pre-defined values, such as biosample_disease.association_type or subject.granularity. Reference tables containing the allowed values for these fields, which were originally published on the CFDE-CC Documentation Wiki, are linked below:

• All reference tables
  • disease_association_type
  • phenotype_association_type
  • subject_ethnicity
  • subject_granularity
  • subject_race
  • subject_role
  • subject_sex

Submission Prep Script

Sourced from the CFDE-CC Documentation Wiki

Datapackage Submission

As an optional but recommended step before submitting your data package, you may validate your pipeline using either the c2m2-frictionless Python package (see Resources) or by following the steps below for using the frictionless validator, from the CFDE-CC Documentation Wiki Quickstart:

pip install frictionless
If that command fails try:
pip install frictionless-py
Once it's installed, run it by doing:
frictionless validate PATH/TO/JSON_FILE_IN_DIRECTORY
This command takes several minutes to run, and dumps the results into your terminal by default. To make a nicer file to review do:
frictionless validate PATH/TO/JSON_FILE_IN_DIRECTORY > report.txt

Currently, the CFDE Workbench Data Portal accepts complete datapackage submissions in ZIP file format (*.zip).

For specific instructions on using the submission system, see the Contribution Guide.

To submit a datapackage, navigate to the Submission System.

Tutorial

For the April 2022 CFDE Cross-Pollination meeting, the LINCS DCC demonstrated a Jupyter Notebook tutorial on building the file, biosample, and subject tables for LINCS L1000 signature data. The code and files can be found at the following link:

LINCS C2M2 Demo (04-05-2022 Cross-Pollination)

Code snippets from this tutorial corresponding to each step are reproduced below. Note that the C2M2 datapackage building process will vary across DCCs, depending on the types of generated data, ontologies, and access standards in place. In general, the process will be as follows:

1. Become familiar with the current structure of the C2M2, including the required fields across the entity and association tables, and download the most recent version of the JSON schema. Gather any metadata mapping files you may need.

2. Identify the relevant namespace for all files, and build the id_namespace and dcc tables first.

   • For LINCS, the namespace "https://lincsproject.org" is used. The following code will generate the LINCS id_namespace table:

   pd.DataFrame(
     [
       [
         'https://www.lincsproject.org', # id
         'LINCS', # abbreviation
         'Library of Integrated Network-Based Cellular Signatures', # name
         'A network-based understanding of biology' # description
         ]
     ], 
     columns=['id', 'abbreviation', 'name', 'description']
   ).to_csv('id_namespace.tsv', sep='\t', index=False)
   

3. Identify all relevant projects and their associated files that will be included in the C2M2 datapackage. Generate container entity tables (project, collection) that describe logic, theme, or funding-based groups of the core entities. Note that projects and collections may be nested, in the project_in_project and collection_in_collection tables.

   • In the LINCS tutorial, the data comes from the 2021 release of the LINCS L1000 Connectivity Map dataset. In creating a project representing the files from this dataset, there must also be an overarching root project for all LINCS data.

   pd.DataFrame(
     [
       [ 
         'https://www.lincsproject.org', # id_namespace
         'LINCS', # local_id
         'https://www.lincsproject.org', # persistent_id
         date(2013, 1, 1), # creation_time
         'LINCS', # abbreviation
         'Library of Integrated Network-Based Cellular Signatures', # name
         'A network-based understanding of biology' # description
       ], 
       [
         'https://www.lincsproject.org', # id_namespace
         'LINCS-2021', # local_id
         'https://clue.io/data/CMap2020#LINCS2020', # persistent_id
         date(2020, 11, 20), # creation_time
         'LINCS-2021', # abbreviation
         'LINCS 2021 Data Release', # name
         'The 2021 beta release of the CMap LINCS resource' # description
       ]
     ], 
     columns=[
       'id_namespace', 'local_id', 'persistent_id', 'creation_time', 
       'abbreviation', 'name', 'description'
     ]
   ).to_csv('project.tsv', sep='\t', index=False)
   
   pd.DataFrame(
     [
       [
         'https://www.lincsproject.org', # parent_project_id_namespace
         'LINCS', # parent_project_local_id
         'https://www.lincsproject.org/', # child_project_id_namespace
         'LINCS-2021' # child_project_local_id
       ]
     ],
     columns=[
       'parent_project_id_namespace', 'parent_project_local_id',
       'child_project_id_namespace', 'child_project_local_id'
     ]
   ).to_csv('project_in_project.tsv', sep='\t', index=False)
   

4. Determine the relationships between files and their associated samples or biological subjects, as well as all relevant assay types, analysis methods, data types, file formats, etc. Also identify all appropriate ontological mappings, if any, corresponding to each value from above.

   • The LINCS DCC includes internal drug, gene, and cell line identifiers, which were mapped to PubChem, Ensembl, and Disease Ontology/UBERON manually ahead of time, but other DCCs may already make use of the CFDE-supported ontologies.
   • For instance, the LINCS signature L1000_LINCS_DCIC_ABY001_A375_XH_A16_lapatinib_10uM.tsv.gz comes from the biosample ABY001_A375_XH_A16_lapatinib_10uM (in this case an experimental condition) and the subject cell line A375; has a data type of expression matrix (data:0928); is stored as a TSV file format (format:3475) with GZIP compression format (format:3989); and has a MIME type of text/tab-separated-values.
   • The ABY001_A375_XH_A16_lapatinib_10uM biosample was obtained via the L1000 sequencing assay type (OBI:0002965); comes from a cell line derived from the skin (UBERON:0002097); and was treated with the compound lapatinib (CID:208908).

5. Either manually or programmatically, generate each data table, starting with the core entity tables (file, biosample, subject). This step will depend entirely on the format of a DCC's existing metadata and ontology mapping tables.

6. Generate the inter-entity linkage association tables (file_describes_biosample, file_describes_subject, biosample_from_subject).

   • In the LINCS tutorial, since the filenames come directly from the biosamples, once the file and biosample tables have both been built, file_describes_biosample can be generated.

   fdb = file_df[['id_namespace', 'local_id']].copy()
   fdb = fdb.rename(
     columns={
       'id_namespace': 'file_id_namespace', 
       'local_id': 'file_local_id'
     }
   )
   fdb['biosample_id_namespace'] = fdb['file_id_namespace']
   fdb['biosample_local_id'] = fdb['file_local_id'].apply(file_2_biosample_map_function)
   

7. Assign files, biosamples, and subjects to any collections, if applicable, using the file_in_collection, subject_in_collection, and biosample_in_collection tables.

   • Collections are optional, and can represent files from the same publications or other logical groupings outside of funding.

8. Use provided C2M2 submission preparation script and ontology support files to automatically build term entity tables from your created files.

9. Optionally validate the final datapackage containing all files and the schema using one of the following validator tools:

   • Frictionless validator
   • c2m2-frictionless-datapackage code package

10. Compress the entire directory into a *.zip file and upload to the CFDE Workbench Metadata and Data Submission System.

Return to Documentation