Skip to content

cmem_client.repositories.datasets¤

Repository for managing datasets in Corporate Memory.

Provides DatasetsRepository for listing datasets across projects and for creating, reading, updating and deleting a dataset inside a project. Datasets are addressed by their project and dataset ID, and the available dataset types are described by the dataset plugins.

Examples:

Inspect the dataset types the deployment offers:

>>> from cmem_client.client import Client
>>> client = Client.from_env()
>>> sorted(client.datasets.get_dataset_plugins())
>>> client.datasets.get_plugin_schema("csv")

Create a dataset in a project and read it back:

>>> from cmem_client.models.dataset import Dataset
>>> client.datasets.create_item(
...     Dataset(
...         id="customers",
...         project_id="my-project",
...         data={"type": "csv", "parameters": {"file": "customers.csv"}},
...     )
... )
>>> client.datasets.get_item("my-project", "customers")

Classes:

DatasetDeleteConfig¤

Bases: DeleteConfig

Dataset deletion configuration.

Attributes:

  • model_config –

DatasetsRepository¤

Bases: TaskSearchRepository, DeleteItemProtocol

Repository for datasets.

Functions:

  • create_item – Create a new dataset in a project.
  • delete_all – Delete all items from the repository
  • delete_item – Delete an item from the repository
  • fetch_data – Fetch a list from the DI task search endpoint via a type adapter.
  • get_dataset_plugins – Get all available dataset plugins.
  • get_file_resource – Return a streaming context manager for downloading a file resource.
  • get_item – Get full dataset details including configuration parameters.
  • get_plugin_schema – Get the schema description of a specific task plugin.
  • get_task – Get full task details from the API.
  • items – Get the items of the repository
  • keys – Get the keys of the repository
  • post_file_resource – Upload a file as the resource of a dataset.
  • update_item – Update the configuration of an existing dataset.
  • values – Get the values of the repository

Attributes:

  • logger (Logger) – Gets the client logger

create_item¤

create_item(item)

Create a new dataset in a project.

Parameters:

  • item (Dataset) – Dataset model with project_id, id, data (type, parameters, read_only, uri_property) and optionally metadata.

Returns:

  • Dataset – Created dataset as a validated Dataset model.

Raises:

  • HTTPStatusError – If the creation request fails.

delete_all¤

delete_all()

Delete all items from the repository

delete_item¤

delete_item(key, skip_if_missing=False, configuration=None)

Delete an item from the repository

Parameters:

  • key (str) – The key of the item to delete
  • skip_if_missing (bool) – If True, it is ignored if the deleted item even exists
  • configuration (DeleteItemConfig) – Optional configuration for deletion

Raises:

fetch_data¤

fetch_data()

Fetch a list from the DI task search endpoint via a type adapter.

get_dataset_plugins¤

get_dataset_plugins()

Get all available dataset plugins.

Returns:

  • dict[str, DatasetPlugin] – Dictionary mapping plugin IDs to their plugin descriptions.

Raises:

  • HTTPStatusError – If the request fails.

get_file_resource¤

get_file_resource(project_id, file_name)

Return a streaming context manager for downloading a file resource.

Parameters:

  • project_id (str) – The project ID.
  • file_name (str) – The file resource name or path within the project.

Returns:

  • AbstractContextManager[Response] – A context manager that yields an httpx.Response with streaming access.
  • AbstractContextManager[Response] – Use response.iter_bytes() inside the with block to read chunks.
Example

from pathlib import Path from cmem_client.client import Client client = Client.from_env() client.files.import_item( … path=Path(“customers.csv”), key=”my-project:customers.csv” … ) with client.datasets.get_file_resource(“my-project”, “customers.csv”) as response: … response.raise_for_status() … with Path(“copy.csv”).open(“wb”) as file: … for chunk in response.iter_bytes(): … file.write(chunk) client.files.delete_item(“my-project:customers.csv”)

get_item¤

get_item(project_id, dataset_id)

Get full dataset details including configuration parameters.

Parameters:

  • project_id (str) – The project ID.
  • dataset_id (str) – The dataset ID.

Returns:

  • Dataset – Dataset model with full details including parameters and metadata.

Raises:

  • HTTPStatusError – If the dataset is not found or request fails.

get_plugin_schema¤

get_plugin_schema(plugin_id)

Get the schema description of a specific task plugin.

Parameters:

  • plugin_id (str) – The plugin ID (e.g. csv, json, eccencaDataPlatform).

Returns:

Raises:

  • HTTPStatusError – If the plugin is not found or the request fails.

get_task¤

get_task(project_id, task_id, with_labels=True)

Get full task details from the API.

Parameters:

  • project_id (str) – The project ID.
  • task_id (str) – The task ID.
  • with_labels (bool) – Whether to include labels in the response.

Returns:

  • TaskResponse – The full task details as a TaskResponse model.

items¤

items()

Get the items of the repository

keys¤

keys()

Get the keys of the repository

logger¤

logger: logging.Logger

Gets the client logger

post_file_resource¤

post_file_resource(project_id, dataset_id, file_resource)

Upload a file as the resource of a dataset.

If the dataset resource already exists, uploading a new file replaces it.

Parameters:

  • project_id (str) – The project ID.
  • dataset_id (str) – The dataset ID.
  • file_resource (BinaryIO) – An open binary file object to upload.

Raises:

  • HTTPStatusError – If the upload request fails.

update_item¤

update_item(item)

Update the configuration of an existing dataset.

Parameters:

  • item (Dataset) – Dataset model with project_id, id, and updated data (type, parameters, read_only, uri_property) and metadata.

Raises:

  • HTTPStatusError – If the update request fails.

values¤

values()

Get the values of the repository

Comments