> For the complete documentation index, see [llms.txt](https://help.bioinsight.illumina.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.bioinsight.illumina.com/connected-analytics/api-access/api-concepts/files-and-storage.md).
# Files and Storage
## Analysis Storage
**Analysis storage** is used to determine the required storage size for your data and results. If this size is too small, the workflow will fail because of insufficient storage space, if it is too large, you are charged for storage which is not needed. In general, it is best to use the settings advised by the workflow designer.
For more details, use the [API Reference](https://ica.illumina.com/ica/api/swagger/index.html).
1. To obtain the analysis storage, perform `GET /api/analysisStorages` to obtain the list of available storages and copy the id of the size which you want to use for your pipeline.
2. Platform Core determines the best matching activation code.
3. Do `POST /api/projects/{projectId}/analysis:cwl` or `POST /api/projects/{projectId}/analysis:nextflow` with the obtained value for analysis storage.
## File Output Behavior
For [advanced output mappings](/connected-analytics/project-web-ui/p-flow/f-analyses.md#analysis-output-mappings), the overwrite behavior for file output can be set with the API parameter actionOnExist. This is a string, which supports the following options if a file or folder by that name already exists at that location.
* **Overwrite** (default) : The existing file is overwritten.
* **Rename** : The file is renamed by appending an incremental counter, before the extension.
* **Skip** : The new file will not be uploaded.
## Using Data from your own S3 Storage
{% hint style="info" %}
Only individual files are supported as input data from your own S3 storage.
{% endhint %}
Prerequisite : have your S3 bucket and credentials ready *(public S3 buckets without credentials are not supported)*
1. **If you do not have credentials**, then from the API perform **POST /api/storageCredentials** to store the storage credentials of your S3 bucket. The response will indicate the **id** (uuid) of the credentials.
2. **If you already have credentials which you want to use**, perform a **GET /api/storageCredentials** to retrieve the list of storage credentials and note the **id** (uuid) which has been assigned to the credentials.
3. From the API perform a **POST /api/projects/{projectId}/analysis:cwl** (or Nextflow) with the required parameters for your analysis and the values below
| *CreateCwlAnalysis > CwlanalysisInput > CwlAnalysisStructuredInput > AnalysisDataInput > AnalysisInputExternalData > url : s3://your\_S3\_address* |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| *CreateCwlAnalysis > CwlanalysisInput > CwlAnalysisStructuredInput > AnalysisDataInput > AnalysisInputExternalData > type : pattern: s3* |
| *CreateCwlAnalysis > CwlanalysisInput > CwlAnalysisStructuredInput > AnalysisDataInput > AnalysisInputExternalData > AnalysisS3DataDetails > storageCredentialsId:* uuid from step 2 |
| CreateCwlAnalysis > CwlanalysisInput > CwlAnalysisStructuredInput > AnalysisDataInput > AnalysisInputExternalData > mountPath : location where the input file will be located on the machine that is running the pipeline.
Preferably use a relative path, though an absolute path is allowed. The path must end with the file name which may differ from the original input data.
|
## Duplicate Files on SampleCreationBatch
When running `POST /api/projects/{projectId}/sampleCreationBatch`, the input is checked to see if there are any duplicate files. The file in the lowest-level folder will be considered as the original file and duplicates in higher-level folders or the root folder will be ignored as input. Suppose you have the files `main\file1`, `main\file2` and `main\folder1\file1`. Then, the input will be `main\file2` because it has no duplicate, and `main\folder1\file1` because, even though it is a duplicate file, this instance is in the lowest level folder. `main\file1` will not be used because it is a duplicate in a higher-level folder.
## Optimizing File Upload Session Speed
The following API endpoints are used to initiate and complete a file upload session:
* [Endpoint](https://ica.illumina.com/ica/api/swagger/index.html#/Project%20Data/createFolderUploadSession) `/api/projects/{projectId}/data/{dataId}/folderUploadSessions` is used to start a file upload session.
* [Endpoint](https://ica.illumina.com/ica/api/swagger/index.html#/Project%20Data/completeFolderUploadSession) `/api/projects/{projectId}/data/{dataId}/folderUploadSessions/{folderUploadSessionId}:complete` is used to signal completion of the upload. In the request body is the parameter `numberOfExpectedUploadedFiles` which is used to ensure Platform Core and S3 objects are in sync. Set this parameter as follows:
* Pass the **number of files that were uploaded** in the folder session for fast verification of all files being accounted for.
* If no count is available, set this value to **-1**.
| Phase | Folder Session Status |
|---|
| Files are being uploaded. | Open |
| Call folderUploadSessions completion endpoint. | Closed |
| Platform Core compares the file count to the numberOfExpectedUploadedFiles and all files are accounted for. | Completed (fast) |
| If no numberOfExpectedUploadedFiles value is provided, or if the number of files in the folder doesn't match the specified number, the system needs to verify that every file has been stored and indexed | Completed (slower) |
A folder upload session will still transition to Completed even if the number of files found does not match the numberOfExpectedUploadedFiles specified. This is because any folder upload session may only capture a portion of the upload to a folder. There may be files already present when the folder upload was started, or multiple concurrent folder upload sessions to the same folder tree. For this reason, the **numberOfExpectedUploadedFiles** parameter **should be treated as an optimization** hint.
{% hint style="info" %}
You can also manually verify if your files are present with the `get /api/projects/{projectId}/data/{dataId}/folderUploadSessions/{folderUploadSessionId}` endpoint.
{% endhint %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://help.bioinsight.illumina.com/connected-analytics/api-access/api-concepts/files-and-storage.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.