> For the complete documentation index, see [llms.txt](https://help.bioinsight.illumina.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.bioinsight.illumina.com/connected-insights/configure/configuration/c-pipeline-and-qc-configuration/pipeline-and-qc-yaml-reference.md). # Pipeline and QC Configuration YAML This topic documents the **complete** YAML schema for a custom pipeline configuration file — the `pipeline:` registration block, the `threshold:` QC section, and how they relate to the file-layout sections. For file-layout sections (`sampleSheet`, `jointFiles`, `sampleFiles`), refer to [Custom Pipeline Configuration](/connected-insights/configure/configuration/du-introduction/du-custom-pipeline-configuration.md). For uploading the file in Connected Insights, refer to [Pipeline and QC Configuration](/connected-insights/configure/configuration/c-pipeline-and-qc-configuration.md). ## Schema Overview The file has three logical parts. The `pipeline:` and `threshold:` blocks are required. The file-layout sections are what make the Results, Visualize, and Lab QC tabs populate, and in practice at least the `jointFiles` section is needed: it declares the variant files (SNV/CNV/SV/fusion), and an analysis imported without it brings in no variants and may fail. The `sampleSheet` and `sampleFiles` sections are genuinely optional. | Part | YAML key(s) | Purpose | Required? | | ---------------------------- | ------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | **1. Pipeline registration** | `pipeline:` | Names the pipeline, sets version and type, and (optionally) the Platform Core pipeline id | Yes | | **2. File layout** | `sampleSheet:` `jointFiles:` `sampleFiles:` | Tells Connected Insights where each analysis output file lives | `jointFiles` needed to import variants; `sampleSheet` / `sampleFiles` optional | | **3. QC thresholds** | `threshold:` | QC metrics, pass/fail bounds, Results up-leveling, gene-coverage line | Yes (≥ 1 metric) | ### Conventions Used Throughout the File * **Path placeholders** are substituted per case at ingestion time: | Placeholder | Resolves to | | ----------------------- | ---------------------------------- | | `{sampleId.DNA}` | the DNA sample's `Sample_ID` | | `{sampleId.RNA}` | the RNA sample's `Sample_ID` | | `{sampleId.tumor.DNA}` | the tumor DNA `Sample_ID` | | `{sampleId.normal.DNA}` | the normal DNA `Sample_ID` | | `{pairId}` | the `Pair_ID` that groups the case | | `{prefix}` | the output file prefix column | * **`, Optional` suffix** — a file path ending in `, Optional` may be absent without failing ingestion. Omit the suffix to require the file. * **Ordered candidate lists** — every file slot accepts a **list** of candidate patterns. Connected Insights uses the **first pattern that matches**, so several naming variants can be listed for the same logical file. * **Unknown keys are ignored** — Connected Insights silently skips keys it does not recognize, so extra annotation in the file is harmless. ## `pipeline:` — Pipeline Registration The `pipeline:` block registers the pipeline in Connected Insights. `displayName`, `name`, and `version` are required; everything else is optional. ```yaml pipeline: displayName: Sample Custom Pipeline v1.0.0 name: Sample_Custom_Pipeline version: 1.0.0 icaPipelineId: 00000000-0000-0000-0000-000000000000 icaUrn: urn:ilmn:ica:pipeline:00000000-0000-0000-0000-000000000000 type: WGS # only "WGS" is meaningful (selects the DRAGEN .csv QC parser); omit for TSO 500-style MetricsOutput.tsv successMarkerFile: - "{prefix}/{prefix}.time_metrics.csv, Optional" failureMarkerFile: "failure.txt" ``` | Field | Required | Value / notes | | ------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `displayName` | Yes | Human-readable name shown in Connected Insights. Use a report-friendly name. | | `name` | Yes | Internal pipeline name. For pipelines used in manual upload, use only letters, numbers, underscores, and dashes (no spaces or special characters). | | `version` | Yes | Pipeline version string (e.g. `1.0.0`). | | `icaPipelineId` | No | Platform Core pipeline UUID. Provide **together with** `icaUrn` to link the pipeline to Platform Core for cloud import. The pipeline remains usable via the Data Uploader with or without these ids. See the pairing rule below. | | `icaUrn` | No | Platform Core pipeline URN (`urn:ilmn:ica:pipeline:`). Provide together with `icaPipelineId`. | | `type` | No | Connected Insights only acts on the value `WGS`, which selects the DRAGEN-style `.csv` QC-metrics parser. Set `type: WGS` when the QC metrics are DRAGEN `.csv` files; **omit `type` entirely** when they are a TSO 500-style `MetricsOutput.tsv`. Any other string has no effect (treated as TSO 500 style). See [`threshold:` — QC Metrics](#threshold-qc-metrics). | | `successMarkerFile` | No | One or more glob patterns whose presence marks a **completed** run. **Data Uploader only** — see note below. | | `failureMarkerFile` | No | File name whose presence marks a **failed** run. **Data Uploader only** — see note below. | ### Platform Core Ids: Pairing Rule and Ingestion Paths `icaPipelineId` and `icaUrn` must be supplied **together or not at all**: * **Both present** → the pipeline is **also** linked to Platform Core and can be auto- or manually imported from there. It **still** supports Data Uploader and manual local upload. * **Both absent** → no Platform Core link; the pipeline is used via the Data Uploader and manual local upload only. * **Only one present** → the upload is **rejected** with a validation error. ### Marker Files Apply to Folder Monitoring Only `successMarkerFile` and `failureMarkerFile` are used **only by the Connected Insights Data Uploader when it monitors a local folder** for auto-upload — they signal to the uploader when a run in that folder has finished (success) or failed. They are **ignored for Platform Core cloud analysis**, which reports completion through the platform's own analysis-status events. Omit both when relying solely on Platform Core analysis status for completion. ## `threshold:` — QC Metrics The `threshold:` section defines the QC metrics Connected Insights evaluates for cases from this pipeline. At least **one** metric is required. Each metric is described once, with both its display properties and its pass/fail bounds: ```yaml threshold: defaultCategory: SAMPLE_QC metrics: - name: "Total Reads (Count)" group: "MAPPING/ALIGNING SUMMARY" raw: "Total input reads" displayInOverView: true type: RANGE isLowerEnable: true isUpperEnable: false lowerValue: "1000000" ``` | Field | Meaning | | ------------------------------------ | -------------------------------------------------------------------------------------------- | | `defaultCategory` | Category applied to every metric that doesn't set its own `category`. Typically `SAMPLE_QC`. | | `metrics[].name` | Display label for the metric. | | `metrics[].group` | Display section the metric is grouped under. | | `metrics[].raw` | Maps the metric to the underlying lab/DRAGEN metric. See "The `raw` mapping" below. | | `metrics[].category` | Overrides `defaultCategory` for this metric: `RUN_QC`, `SAMPLE_QC`, or `GENE_QC`. | | `metrics[].type` | Rule type: `RANGE` (numeric / percentage / count) or `BOOLEAN`. | | `metrics[].isLowerEnable` | `true` to evaluate the lower bound (lower spec limit). | | `metrics[].isUpperEnable` | `true` to evaluate the upper bound (upper spec limit). | | `metrics[].lowerValue` | Lower bound value, as a string. Omit if the lower bound is disabled. | | `metrics[].upperValue` | Upper bound value, as a string. Omit if the upper bound is disabled. | | `metrics[].displayInOverView` | `true` to up-level this metric to the case **Results** tab and report. | | `metrics[].disableDisplayInOverView` | `true` to force-hide the metric from the Results tab. | ### Pass / Fail / Unspecified QC evaluation follows the Connected Insights rule shown in the UI: > If a lower, an upper, or both limits are defined, the corresponding QC metric is marked **Pass** or **Fail**. If no limits are specified, it is marked **Unspecified**. So a metric with `isLowerEnable: true` and a `lowerValue` receives a Pass/Fail verdict; a metric with neither bound enabled is carried as **Unspecified** (informational only). ### The QC Metrics File The metrics Connected Insights evaluates come from the file(s) declared in `jointFiles.metricsQCFile`. A custom pipeline must emit that file in one of the two formats Connected Insights parses. Each `threshold` metric then points at a row in that file through its `raw` mapping. For the `metricsQCFile` key and other file-layout keys, refer to [Custom Pipeline Configuration — Joint Files](/connected-insights/configure/configuration/du-introduction/du-custom-pipeline-configuration.md#joint-files). Every metric is either a **run-level** metric (applies to the whole sequencing run — e.g. percent reads passing filter, percent Q30) or a **sample-level** metric (applies to one sample — e.g. coverage, contamination). The two formats mark that distinction differently, as described below. Run-level metrics are configured with `category: RUN_QC`; sample metrics with `SAMPLE_QC` (or `GENE_QC`). **TSO 500 style — `MetricsOutput.tsv` (recommended).** A single tab-separated file divided into bracketed sections. The run-level metrics live in the `[Run QC Metrics]` section; each remaining bracketed section (e.g. `[DNA Library QC Metrics]`) holds sample metrics. Every section starts with a header row, then one row per metric: the metric name, optional `LSL Guideline` / `USL Guideline` columns, then the value — a single `Value` column for the run section, or one value column per sample for a sample section. ``` [Run QC Metrics] Metric (UOM) LSL Guideline USL Guideline Value PCT_PF_READS (%) 80.0 NA 94.7 PCT_Q30_R1 (%) 80.0 NA 94.2 [DNA Library QC Metrics] Metric (UOM) LSL Guideline USL Guideline Sample_A Sample_B CONTAMINATION_SCORE (NA) 0 3106 12 18 MEDIAN_EXON_COVERAGE (Count) NA NA 155 142 ``` The `[Header]` and `[Analysis Details]` sections are read as **run metadata** (a set of `Field Value` rows) and shown as run/analysis information. The field name is a fixed, closed set — Connected Insights recognizes only the labels below (case-insensitive), and any other row in these sections is ignored: | Field label (either section) | Shown as | | ------------------------------ | ---------------- | | `Run ID` | Run ID | | `Run Name` | Run Name | | `Workflow Version` | Workflow Version | | `Module Version` | Module Version | | `Pipeline Version` | Pipeline Version | | `Output Date` or `Report Date` | Output Date | | `Output Time` or `Report Time` | Output Time | These are metadata only, not QC thresholds; only the `[Run QC Metrics]` and sample sections carry the metrics matched by the `threshold:` config. ``` [Header] Output Date 2026-01-10 Pipeline Version 2.5.2 [Analysis Details] Run Name MyRun_2026-01-10 Workflow Version 4.5.0 ``` **DRAGEN style — individual `*.csv` files.** One or more comma-separated files (`mapping_metrics.csv`, `wgs_coverage_metrics.csv`, …), each row being `Section,Group,Metric,Value[,Percentage]`. The first column is the section name. Here the run-level metrics are the rows whose section is `Run QC Metrics`; all other rows are sample metrics. An optional 5th column is exposed as a separate ` (Percentage)` metric. To use this format, set `pipeline.type: WGS` so Connected Insights selects the DRAGEN parser. ``` MAPPING/ALIGNING SUMMARY,Sample_A,Total input reads,395109246,100.00 MAPPING/ALIGNING SUMMARY,Sample_A,Estimated sample contamination,0.008 ``` (Typical DRAGEN `*_metrics.csv` files contain only sample sections; run-level rows are uncommon in that format.) Choose one style. The TSO 500 `MetricsOutput.tsv` is preferred — its sections and per-sample columns are self-describing, which makes the `raw` mapping easier to author. ### The `raw` Mapping `raw` connects a `threshold` metric to a specific row in the QC file. Connected Insights matches (case-insensitively): * Scalar `raw` or `raw.name` → the **metric name** in the file (the first column of the metric row). * Source section for binding → `raw.group` when set in a nested object; otherwise the top-level `group` on the metric. The metric's `category` selects which part of the file is searched: `RUN_QC` metrics are matched against the run-level section by metric name only; `SAMPLE_QC` (and `GENE_QC`) metrics are matched within a section by metric name **and** source section. `raw` can be written three ways: * **Omit `raw`** when the display `name` already equals the metric name in the file. * **Scalar string (default)** — the file's metric name when the display `name` differs and the UI `group` matches the metrics-file section: ```yaml - name: "Total Reads (Count)" group: "MAPPING/ALIGNING SUMMARY" raw: "Total input reads" ``` * **Nested object (optional)** — when the UI display `group` should differ from the source section, or when the same metric name appears in more than one source section: ```yaml raw: name: "Q30 bases (Percentage)" group: "TUMOR MAPPING/ALIGNING SUMMARY" ``` **Worked example (TSO 500 style).** Given the `[DNA Library QC Metrics]` section above, this metric surfaces `CONTAMINATION_SCORE` on the Results tab as "Estimated Sample Contamination" with an upper limit of 3106. The display `group` is shortened for the UI; `raw.group` points at the source section: ```yaml - name: "Estimated Sample Contamination" # display label on the Results tab group: "DNA Library QC" # display grouping category: SAMPLE_QC raw: name: "CONTAMINATION_SCORE (NA)" # metric name in the file group: "DNA Library QC Metrics" # section name in the file type: RANGE isLowerEnable: false isUpperEnable: true upperValue: "3106" ``` ### QC Categories | Category | Use | | ----------- | ------------------------------------------------------ | | `RUN_QC` | Run-level metrics (e.g. percent reads passing filter) | | `SAMPLE_QC` | Per-sample metrics (coverage, contamination, Q30, …) | | `GENE_QC` | Per-gene metrics — notably the gene-coverage plot line | ### Setting the Gene-coverage Plot Line To set the reference line on the per-gene coverage plot (shown on the Results / Visualize tabs), add a `GENE_QC` metric named `Gene Coverage`. Its `lowerValue` is the coverage line: ```yaml - name: "Gene Coverage" group: "Coverage Plot" category: GENE_QC raw: "GENE_COVERAGE" type: RANGE isLowerEnable: true isUpperEnable: false lowerValue: "900" disableDisplayInOverView: true ``` This is the supported way to configure the coverage-plot line; it is set here rather than through a separate plot-config field. ## Worked Examples ### Data Uploader Pipeline (No Platform Core Ids) A pipeline ingested via the Data Uploader or manual local upload, with a single small-variant file and one Pass/Fail metric. Platform Core ids omitted — no Platform Core import path. ```yaml pipeline: displayName: My Lab WGS v1.0.0 name: My_Lab_WGS version: 1.0.0 type: WGS jointFiles: snvFiles: - "{sampleId.DNA}/{sampleId.DNA}.hard-filtered.vcf.gz" metricsQCFile: - "{sampleId.DNA}/{sampleId.DNA}.mapping_metrics.csv, Optional" threshold: defaultCategory: SAMPLE_QC metrics: - name: "Average alignment coverage over target region" group: "COVERAGE SUMMARY" displayInOverView: true type: RANGE isLowerEnable: true isUpperEnable: false lowerValue: "100" ``` For `jointFiles` details, refer to [Custom Pipeline Configuration — Joint Files](/connected-insights/configure/configuration/du-introduction/du-custom-pipeline-configuration.md#joint-files). ### Pipeline with Platform Core Linking Adding the paired Platform Core ids links the pipeline to Platform Core for auto- or manual import. The same configuration remains usable via the Data Uploader. Both ids must be present, or neither. ```yaml pipeline: displayName: My Lab WGS v1.0.0 name: My_Lab_WGS version: 1.0.0 icaPipelineId: 4f2b9c10-1234-5678-9abc-def012345678 icaUrn: urn:ilmn:ica:pipeline:4f2b9c10-1234-5678-9abc-def012345678 type: WGS ``` ### Gene-coverage Plot Line A `GENE_QC` metric named `Gene Coverage` whose `lowerValue` sets the coverage reference line: ```yaml - name: "Gene Coverage" group: "Coverage Plot" category: GENE_QC raw: "GENE_COVERAGE" type: RANGE isLowerEnable: true isUpperEnable: false lowerValue: "900" disableDisplayInOverView: true ``` ### Same Metric Under Tumor and Normal Groups Use `raw.group` to point two same-named metrics at their respective source groups: ```yaml - name: "Q30 Bases (Percentage)" group: "TUMOR MAPPING/ALIGNING SUMMARY (TUMOR-NORMAL)" raw: name: "Q30 bases (Percentage)" group: "TUMOR MAPPING/ALIGNING SUMMARY" type: RANGE isLowerEnable: true isUpperEnable: false lowerValue: "80" - name: "Q30 Bases (Percentage)" group: "NORMAL MAPPING/ALIGNING SUMMARY (TUMOR-NORMAL)" raw: name: "Q30 bases (Percentage)" group: "NORMAL MAPPING/ALIGNING SUMMARY" type: RANGE isLowerEnable: true isUpperEnable: false lowerValue: "80" ``` A complete, copy-pasteable file combining all of the above — pipeline block, file layout, and the full QC metric set — is provided in [Sample Custom Pipeline YAML](/connected-insights/configure/configuration/c-pipeline-and-qc-configuration/pipeline-and-qc-yaml-reference/sample-custom-pipeline.md). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://help.bioinsight.illumina.com/connected-insights/configure/configuration/c-pipeline-and-qc-configuration/pipeline-and-qc-yaml-reference.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.