> For the complete documentation index, see [llms.txt](https://help.bioinsight.illumina.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.bioinsight.illumina.com/connected-insights/configure/configuration/du-introduction/du-custom-pipeline-configuration.md).
# Custom Pipeline Configuration
## Custom Pipeline Configuration
The custom pipeline option is designed to make Connected Insights understand the structure of the secondary analysis output files produced by a pipeline that is not yet compatible with the software. This option also requires the creation of a workflow schema file that describes the content and location of the secondary analysis output files. For an example of a how to configure a custom pipeline for TSO 500 Analysis Module v2.2, refer to [Custom Pipeline Configuration Example](#custom-pipeline-configuration-example).
In Connected Insights v5.5 and later, the same YAML file also **registers the pipeline and its QC thresholds** when uploaded from **Configuration > General > Pipeline & QC Configuration**. For pipeline registration (`pipeline:`), QC thresholds (`threshold:`), and uploading the file, refer to [Pipeline and QC Configuration](/connected-insights/configure/configuration/c-pipeline-and-qc-configuration.md) and [Pipeline and QC Configuration YAML](/connected-insights/configure/configuration/c-pipeline-and-qc-configuration/pipeline-and-qc-yaml-reference.md). The sections below document the **file-layout** portions of the YAML (`sampleSheet`, `jointFiles`, `sampleFiles`).
### Select Custom Pipeline
1. Input secondary analysis output folder to be monitored.
2. In Configuration Settings, select the radio button next to Configure custom pipeline.
3. If necessary, create a workflow schema file by selecting **Download the template file**. For more information on setting up the template file, refer to [Create a Workflow Schema File](#create-a-workflow-schema-file)
4. Select **Choose File** to upload your template file.
5. For Custom Pipeline Name, enter a name for the pipeline.
6. For Test Definition, select the applicable definition.
7. \[Optional] Input the location of the folder where [case metadata data](/connected-insights/configure/configuration/du-introduction/du-custom-case-dataupload.md) is available.
8. Select **Save**.
### Create a Workflow Schema File
To set up data upload for secondary analysis output data that is not yet compatible with Connected Insights, create a workflow schema file (.yaml format). This file specifies the files in the secondary analysis output data that Connected Insights analyzes. This file is only used when configuring a custom pipeline through **Data Upload > From Local Storage**.
Download a workflow schema file template from Connected Insights as follows.
1. On the top toolbar, select **Configuration**.
2. Select the **General** tab.
3. Select **Data Upload**.
4. Select **From Local Storage**.
5. For Define and Monitor Data Uploads, select **Add Path**.
6. For Configuration Settings, select the radio button next to Configure custom pipeline.
7. Select **Download a template file** to download the workflow schema template file.\
If you do not want to create a pipeline, select **Cancel**. When prompted, select **Yes, clear**.
8. Edit the file as needed to reflect the files for upload. Refer to the following topics that pertain to the workflow schema template file sections:
* [Pipeline and QC Configuration YAML](/connected-insights/configure/configuration/c-pipeline-and-qc-configuration/pipeline-and-qc-yaml-reference.md) — `pipeline:` registration and `threshold:` QC metrics (required when uploading from **Pipeline & QC Configuration**)
* [Sample Sheet](#sample-sheet)
* [Joint Files](#joint-files)
* [Sample Files](#sample-files)
If , Optional is after the file name, then Connected Insights uploads the file if it is available or moves on to the next available file.
DNA\_Control
RNA\_Control
| DNA
RNA
|
| Lung\_001 | DNA and RNA | Non-small cell lung cancer (SNOMEDCT ID 254637007) | Lung\_DNA\_001
Lung\_RNA\_001
| DNA
RNA
|
| Breast\_002 | DNA | Malignant tumor of breast (SNOMEDCT ID 254837009) | Breast\_DNA\_002 | DNA |
### Example Analysis Results
Open the secondary analysis results folder and find the files that must be identified in the workflow schema file. The following example shows the secondary analysis results folder structure:
### Example Workflow Schema File
For more information, refer to Create a Workflow Schema File in [Custom Pipeline Configuration](#custom-pipeline-configuration).
The following example shows the workflow schema file structure:
```yaml
pipeline:
successMarkerFile: Logs_Intermediates/MetricsOutput/MetricsOutput.tsv
failureMarkerFile: "./failure.txt"
sampleSheet:
filePath: SampleSheet.csv
jointFiles:
purityPloidyFiles: Logs_Intermediates/CombinedVariantOutput/{pairId}/{pairId}_CombinedVariantOutput.tsv, Optional
gisFile: Logs_Intermediates/Gis/{sampleId.DNA}/{sampleId.DNA}.gis.json, Optional
msiFile: Logs_Intermediates/Msi/{sampleId.DNA}/{sampleId.DNA}.msi.json, Optional
tmbFile: Logs_Intermediates/Tmb/{sampleId.DNA}/{sampleId.DNA}.tmb.json, Optional
lohFiles:
- Logs_Intermediates/Gis/{sampleId.DNA}/{sampleId.DNA}.abcn_genes.tsv, Optional
snvFiles: Logs_Intermediates/VariantMatching/{sampleId.DNA}/{sampleId.DNA}_MergedSmallVariants.genome.vcf, Optional
cnvFiles:
- Logs_Intermediates/CnvCaller/{sampleId.DNA}/{sampleId.DNA}_CopyNumberVariants.vcf, Optional
rnaSpliceFiles: Logs_Intermediates/RnaSpliceVariantCalling/{sampleId.RNA}/{sampleId.RNA}_SpliceVariants.vcf, Optional
rnaFusionFiles:
- Logs_Intermediates/FusionCalling/{sampleId.RNA}/{sampleId.RNA}.vcf.gz, Optional
- Logs_Intermediates/RnaFusionMerge/{sampleId.RNA}/{sampleId.RNA}_AllFusions.csv, Optional
metricsQCFile: Logs_Intermediates/MetricsOutput/MetricsOutput.tsv, Optional
sampleFiles:
tumorPairId:
alignmentFiles:
dnaBamFile: Logs_Intermediates/DnaAlignment/{sampleId.DNA}/{sampleId.DNA}.bam, Optional
dnaBaiFile: Logs_Intermediates/DnaAlignment/{sampleId.DNA}/{sampleId.DNA}.bam.bai, Optional
rnaBamFile: Logs_Intermediates/RnaAlignment/{sampleId.RNA}/{sampleId.RNA}.bam, Optional
rnaBaiFile: Logs_Intermediates/RnaAlignment/{sampleId.RNA}/{sampleId.RNA}.bam.bai, Optional
```
{% hint style="warning" %}
If `, Optional` is after the file name, then Connected Insights uploads the file if it is available or moves on to the next available file.
{% endhint %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://help.bioinsight.illumina.com/connected-insights/configure/configuration/du-introduction/du-custom-pipeline-configuration.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.