Custom Workflow Tasks¤
A custom workflow task is an operator that can be used in a workflow.
Intended audience: Linked Data Experts and Domain Experts
| Name | Description |
|---|---|
| Add project files | Adds file resources to the project that are piped into the input port. |
| Build Packages | Build package archives from a CPA manifest, using graphs, projects and files from this instance. |
| Bump Version in Package Manifest | Raise the version of a CPA manifest, by semantic level or to the current date. |
| Cancel Workflow | Cancels a workflow if a specified condition is fulfilled. A typical use case for this operator is to cancel the workflow execution if the input data is empty. |
| Clear dataset | Clears the dataset that is connected to the output of this operator. |
| Combine CSV files | Combine CSV files with the same structure to one dataset. |
| Concatenate to file | Concatenates values into a file. |
| Create Embeddings | Turn the text of each entity into a vector embedding and add it to the entity. |
| Create or Update Graph Insights Snapshots | Create a snapshot for a graph that has none, or update it if one already exists. |
| Create/Update Salesforce Objects | Manipulate data in your organization’s Salesforce account. |
| Delete project files | Removes file resources from the project based on a regular expression. |
| Distinct by | Removes duplicated entities based on user-defined paths. Duplicates can be resolved by keeping the first or last entity, or by keeping the entity with the minimum or maximum value of a compare path. |
| Download CKAN files | Download the files of a dataset from a CKAN service. |
| Download file | Downloads a file from a given URL. |
| Download Git files | Download files from a git repository and hand them to the next task. |
| Download Nextcloud files | Download files from a given Nextcloud instance. |
| Download Office 365 Files | Download files from Microsoft OneDrive or Sites |
| Download S3 objects | Download objects from a bucket in S3 or S3-compatible object storage. |
| Download SSH files | Download files from an SSH server. |
| Evaluate template | Evaluates a template on a sequence of entities. Can be used after a transformation or directly after datasets that output a single table, such as CSV or Excel. A template that uses no input variables can be evaluated without an input. |
| Execute a command in a kubernetes pod | Connect to a cluster, execute a command and gather the output. |
| Execute Catalog Query | Run a query stored in a query catalog and hand on its result. |
| Execute commands via SSH | Execute a command on an SSH server. |
| Execute Instructions | Prompt a large language model once per entity and add the answer to the entity. |
| Execute REST requests | REST operator that fetches and optionally merges data from a REST endpoint. It supports executing multiple requests either via input entities that each overwrite config parameters or via paging. If you only need to download a single file, the “Download file” operator might be the better option. Most features are currently only supported for JSON REST APIs. By default the output of this operator is one entity per response with a single property ‘result’ that holds the response body. Responses of multiple requests are merged into a single JSON array if the ‘Parse JSON’ operator is connected or the Accept header starts with ‘application/json’. Alternatively the response/s can be written to the file of a connected file-based dataset, either as a merged JSON file (JSON dataset) or, if the dataset file ends in ‘.zip’, one file per response inside a ZIP file. |
| Execute Spark function | Applies a specified Scala function to a specified field. |
| Extract from PDF files | Extract text and tables from PDF files |
| Generate base36 IRDIs | Create unique ECLASS IRDIs. |
| Generate random values | Generates entities with random values. |
| Generate SHACL shapes from data | Generate SHACL node and property shapes from a data graph |
| Get Catalog Query | Emit one entity describing a query stored in a query catalog. |
| Get emails | Search IMAP mailbox and get email metadata or file attachments. |
| Get ntfy attachments | Download the attachments of a topic’s messages from an ntfy server. |
| Get ntfy messages | Read the messages of a topic from an ntfy server. |
| Get project files | Get file resources from the project. |
| Get workflow report | Output a workflow execution report as a JSON file. |
| GraphQL query | Sends a GraphQL query or mutation to an endpoint and returns the result as entities. |
| Hetzner Cloud: List Resources | Inventory all existing resources of a Hetzner Cloud project (servers, volumes, networks, load balancers, IPs, firewalls, SSH keys, snapshots, …), one entity per resource in a single union schema. |
| Hetzner Cloud: List Server Types | List all available Hetzner Cloud server (VM) types, one entity per server type and location, including specs and per-location pricing. |
| Install Packages | Install packages and their dependencies from a marketplace service or from package archives. |
| Join tables | Joins a set of inputs into a single table. Expects a list of entity tables and links. All entity tables are joined into the first entity table using the provided links. |
| jq | Process a JSON document with a jq filter / program. |
| JQL query | Search and retrieve Jira issues. |
| JSON to File | Writes a JSON string held in a field on each valid incoming entity to a file. Depending on the output mode, it produces one file per entity, packs all entities into a single ZIP archive, or merges them into a single JSON array file. Produces a file entity downstream, suitable for wiring into a file-backed dataset or any operator that consumes file entities. |
| Kafka Consumer (Receive Messages) | Reads messages from a Kafka topic as entities, or saves them to a dataset (Consumer). |
| Kafka Producer (Send Messages) | Sends entities, or messages from a dataset, to a Kafka topic (Producer). |
| List Catalog Queries | Emit one entity per query stored in a query catalog. |
| List Git files | List the files of a git repository, without transferring their content. |
| List Nextcloud files | List directories and files from a given Nextcloud folder. |
| List Office 365 Files | List files from OneDrive or Sites |
| List project files | List file resources from the project. |
| List S3 objects | List the objects of a bucket in S3 or S3-compatible object storage. |
| List SSH files | List the files of a directory on an SSH server, with their metadata. |
| Merge tables | Stores sets of instance and mapping inputs as relational tables with the mapping as an n:m relation. Expects a list of entity tables and links. All entity tables have a relation to the first entity table using the provided links. |
| Normalize units of measurement | Custom task that will substitute numeric values and pertaining unit symbols with a SI-system-unit normalized representation. |
| OAuth2 Authentication | Provide an OAuth2 access token for other tasks (via config port). |
| Office 365 Upload Files | Upload files to OneDrive or a site Sharepoint |
| Parse JSON | Parses a JSON string held in a field on each incoming entity. Typically used before a transformation task. Takes exactly one input. |
| Parse XML | Takes exactly one input and reads either the defined inputPath or the first value of the first entity as XML document. Then executes the given output entity schema similar to the XML dataset to construct the result entities. |
| Parse YAML | Parses YAML documents from a code field, entities or files and converts them to JSON. |
| Pivot | The pivot operator takes data in separate rows, aggregates it and converts it into columns. |
| Publish Packages | Publish package archives to a marketplace service. |
| Reason | Performs OWL reasoning. |
| Request RDF triples | A task that requests all triples from an RDF dataset. |
| Run Apify actor | Run an Apify actor and deliver its dataset as a file. |
| Scheduler | Executes a workflow at specified intervals. |
| Search addresses | Looks up locations from textual descriptions using the configured geocoding API. Outputs results as RDF. |
| Search for Logs | Search and retrieve logs from a Logpoint SIEM system with flexible schema output. |
| Search LDAP directory | Query LDAP directory services such as OpenLDAP or Active Directory. |
| Search Vector Embeddings | Search for the top-k nearest matches to a query embedding in Postgres Vector Store (PGVector). |
| Select random entities | Select X random entities from an input dataset. |
| Send email | Sends an email using an SMTP server. |
| Send Mattermost messages | Post messages to Mattermost users and channels through a bot account. |
| Send ntfy attachment | Send a file as an attachment to a topic on an ntfy server. |
| Send ntfy message | Send a push notification to a topic on an ntfy server. |
| Set execution variable | Sets an execution variable to the first value of the (single) input and passes the input through unchanged. The variable is written to the ‘execution’ scope and can be read downstream as ‘execution. |
| Set or Overwrite parameters | Connect this task to a config port of another task in order to set or overwrite the parameter values of this task. |
| Set parameters | Set and overwrite parameters of a task. |
| SHACL validation with pySHACL | Performs SHACL validation with pySHACL. |
| SOQL query (Salesforce) | Executes a custom Salesforce Object Query (SOQL) to return sets of data your organization’s Salesforce account. |
| Spark SQL query | Executes a Spark SQL query on the input entities and returns the result as its output. |
| SPARQL Construct query | A task that executes a SPARQL Construct query on a SPARQL enabled data source and outputs the SPARQL result. If the result should be written to the same RDF store it is read from, the SPARQL Update operator is preferable. |
| SPARQL Select query | A task that executes a SPARQL Select query and outputs the SPARQL result. |
| SPARQL Update query | A task that outputs SPARQL Update queries for every entity from the input based on a SPARQL Update template. The output of this operator should be connected to the SPARQL datasets to which the results should be written. |
| Split file | Split a file into multiple parts with a specified size. |
| SQL Update query | A task that outputs SQL queries. The output of this operator should be connected to a remote SQL endpoint on which queries should be executed. |
| Start Workflow per Entity | Loop over the output of a task and start a sub-workflow for each entity. |
| Store Vector Embeddings | Store embeddings into Postgres Vector Store (PGVector). |
| Uninstall Packages | Uninstall packages and their unused dependencies from this instance. |
| Unpivot | Given a list of table columns, transforms those columns into attribute-value pairs. |
| Upload File to Knowledge Graph | Uploads an N-Triples or Turtle (limited support) file from the file repository to a ‘Knowledge Graph’ dataset. The output of this operatorcan be the input of datasets that support graph store file upload, e.g. ‘Knowledge Graph’. The file will be uploaded to the graph specified in that dataset. |
| Upload files to CKAN | Publish files as the files of a dataset on a CKAN service. |
| Upload files to Nextcloud | Upload files to a given Nextcloud instance. |
| Upload files to S3 | Upload files to a bucket in S3 or S3-compatible object storage. |
| Upload Git files | Commit the files of the preceding task into a git repository. |
| Upload local files | Replace a file dataset resource with a local file or upload multiple local files to a project. |
| Upload SSH files | Upload files to an SSH server. |
| Validate Entities | Use a JSON schema to validate entities or a JSON dataset. |
| Validate Knowledge Graph | Use SHACL shapes to validate resources in a Knowledge Graph. |
| Validate OWL consistency | Validates the consistency of an OWL ontology. |
| Validate XML | Validates an XML dataset against a provided XML schema (XSD) file. Any errors are written to the output. Can be used in conjunction with the Cancel Workflow operator in order to stop the workflow if errors have been found. |
| XSLT | A task that converts an XML resource via an XSLT script and writes the transformed output into a file resource. |