Skip to content

You're not viewing the latest version. Click here to go to latest.

documentation.eccenca.com

Remove duplicates

Initializing search

Get a Sandbox

Build
Explore and Author
Consume
Deploy and Configure
Automate
Develop
Release Notes
Tutorials

documentation.eccenca.com

Build
Build
- Introduction to the User Interface
  Introduction to the User Interface
- Rule Operators
  Rule Operators
- Define Prefixes / Namespaces
  Define Prefixes / Namespaces
- Task and Operator Reference
  Task and Operator Reference
  - Aggregators
    
    Aggregators
    
    And
    
    Average
    
    Euclidian distance
    
    First non-empty score
    
    Geometric mean
    
    Handle missing values
    
    Negate
    
    Or
    
    Scale
  - Custom Workflow Tasks
    
    Custom Workflow Tasks
    
    Add project files
    
    Cancel Workflow
    
    Combine CSV files
    
    Concatenate to file
    
    Create Embeddings
    
    Create/Update Salesforce Objects
    
    Delete project files
    
    Distinct by
    
    Download file
    
    Download Nextcloud files
    
    Download Office 365 Files
    
    Download SSH files
    
    Evaluate template
    
    Execute commands via SSH
    
    Execute Instructions
    
    Execute REST requests
    
    Execute Spark function
    
    Extract from PDF files
    
    Generate base36 IRDIs
    
    Generate SHACL shapes from data
    
    Get project files
    
    GraphQL query
    
    Join tables
    
    jq
    
    JQL query
    
    Kafka Consumer (Receive Messages)
    
    Kafka Producer (Send Messages)
    
    List Nextcloud files
    
    List Office 365 Files
    
    List project files
    
    List SSH files
    
    Merge tables
    
    Normalize units of measurement
    
    OAuth2 Authentication
    
    Office 365 Upload Files
    
    Parse JSON
    
    Parse XML
    
    Parse YAML
    
    Pivot
    
    Request RDF triples
    
    Scheduler
    
    Search addresses
    
    Search Vector Embeddings
    
    Send eMail
    
    Send Mattermost messages
    
    Set or Overwrite parameters
    
    SHACL validation with pySHACL
    
    SOQL query (Salesforce)
    
    SPARQL Construct query
    
    SPARQL Select query
    
    SPARQL Update query
    
    Split file
    
    SQL query
    
    Start Workflow per Entity
    
    Store Vector Embeddings
    
    Unpivot
    
    Update SemSpect
    
    Upload File to Knowledge Graph
    
    Upload files to Nextcloud
    
    Upload local files
    
    Upload SSH files
    
    Validate Entities
    
    Validate Knowledge Graph
    
    Validate XML
    
    XSLT
  - Datasets
    
    Datasets
    
    Alignment
    
    Avro
    
    Binary file
    
    CSV
    
    Excel
    
    Excel (Google Drive)
    
    Excel (OneDrive, Office365)
    
    Hive database
    
    In-memory dataset
    
    Internal dataset
    
    Internal dataset (single graph)
    
    JDBC endpoint
    
    JSON
    
    Knowledge Graph
    
    Multi CSV ZIP
    
    Neo4j
    
    ORC
    
    Parquet
    
    RDF file
    
    Snowflake JDBC endpoint
    
    SparkSQL view
    
    SPARQL endpoint
    
    SQL endpoint
    
    Text
    
    XML
  - Distance Measures
    
    Distance Measures
    
    CJK reading distance
    
    Compare physical quantities
    
    Constant similarity value
    
    Cosine
    
    Date
    
    DateTime
    
    Dice coefficient
    
    Geographical distance
    
    Greater than
    
    Inequality
    
    Inside numeric interval
    
    Is substring
    
    Jaccard
    
    Jaro distance
    
    Jaro-Winkler distance
    
    Korean phoneme distance
    
    Korean translit distance
    
    Levenshtein distance
    
    Lower than
    
    Normalized Levenshtein distance
    
    Numeric equality
    
    Numeric similarity
    
    qGrams
    
    Relaxed equality
    
    Soft Jaccard
    
    Starts with
    
    String equality
    
    Substring comparison
    
    Token-wise distance
  - Transformers
    
    Transformers
    
    Combine
    Combine
    
    Concatenate
    
    Concatenate multiple values
    
    Concatenate pairwise
    
    Merge
    
    Conditional
    Conditional
    
    Contains all of
    
    Contains any of
    
    If contains
    
    If exists
    
    If matches regex
    
    Negate binary (NOT)
    
    Conversion
    Conversion
    
    Convert charset
    
    Date
    Date
    
    Compare dates
    
    Current date
    
    Date to timestamp
    
    Duration
    
    Duration in days
    
    Duration in seconds
    
    Duration in years
    
    Number to duration
    
    Parse date pattern
    
    Timestamp to date
    
    Excel
    Excel
    
    Abs
    
    Acos
    
    Acosh
    
    And
    
    Asin
    
    Asinh
    
    Atan
    
    Atan2
    
    Atanh
    
    Avedev
    
    Average
    
    Averagea
    
    Ceiling
    
    Choose
    
    Clean
    
    Code
    
    Combin
    
    Correl
    
    Cos
    
    Cosh
    
    Count
    
    Counta
    
    Covar
    
    Degrees
    
    Devsq
    
    Even
    
    Exact
    
    Exp
    
    Fact
    
    False
    
    Find
    
    Floor
    
    Forecast
    
    Fv
    
    Geomean
    
    If
    
    Int
    
    Intercept
    
    Ipmt
    
    Irr
    
    Large
    
    Left
    
    Ln
    
    Log
    
    Log10
    
    Max
    
    Maxa
    
    Median
    
    Mid
    
    Min
    
    Mina
    
    Mirr
    
    Mod
    
    Mode
    
    Normdist
    
    Norminv
    
    Normsdist
    
    Normsinv
    
    Not
    
    Nper
    
    Npv
    
    Odd
    
    Or
    
    Pearson
    
    Percentile
    
    Percentrank
    
    Pi
    
    Pmt
    
    Poisson
    
    Power
    
    Ppmt
    
    Product
    
    Proper
    
    Pv
    
    Radians
    
    Rand
    
    Rank
    
    Rate
    
    Replace
    
    Rept
    
    Right
    
    Roman
    
    Round
    
    Rounddown
    
    Roundup
    
    Search
    
    Sign
    
    Sin
    
    Sinh
    
    Slope
    
    Small
    
    Sqrt
    
    Standardize
    
    Stdev
    
    Stdeva
    
    Stdevp
    
    Stdevpa
    
    Substitute
    
    Sum
    
    Sumproduct
    
    Sumsq
    
    Sumx2my2
    
    Sumx2py2
    
    Sumxmy2
    
    Tan
    
    Tanh
    
    Tdist
    
    True
    
    Trunc
    
    Var
    
    Vara
    
    Varp
    
    Varpa
    
    Extract
    Extract
    
    Regex extract
    
    Filter
    Filter
    
    Filter by length
    
    Filter by regex
    
    Remove empty values
    
    Remove stopwords
    
    Remove stopwords (remote stopword list)
    
    Remove values
    
    Geo
    Geo
    
    Retrieve coordinates
    
    Retrieve latitude
    
    Retrieve longitude
    
    Linguistic
    Linguistic
    
    Metaphone
    
    Normalize chars
    
    NYSIIS
    
    Soundex
    
    Stem
    
    Metadata
    Metadata
    
    File hash
    
    Input file attributes
    
    Input task attributes
    
    Normalize
    Normalize
    
    Camel case
    
    Capitalize
    
    Clean HTML
    
    Encode URL
    
    Fix URI
    
    Lower case
    
    Remove blanks
    
    Remove duplicates Remove duplicates
    Table of contents
    
    Parameter
    
    Remove parentheses
    
    Remove special chars
    
    Sort words
    
    Strip non-alphabetic characters
    
    Trim
    
    Upper case
    
    Numeric
    Numeric
    
    Aggregate numbers
    
    Compare numbers
    
    Convert Number Base
    
    Extract physical quantity
    
    Format number
    
    Logarithm
    
    Normalize physical quantity
    
    Numeric operation
    
    Numeric reduce
    
    Parser
    Parser
    
    Parse date
    
    Parse float
    
    Parse geo coordinate
    
    Parse geo location
    
    Parse integer
    
    Parse ISIN
    
    Parse SKOS term
    
    Parse string
    
    Replace
    Replace
    
    Excel map
    
    Map
    
    Map with default
    
    Regex replace
    
    Replace
    
    Selection
    Selection
    
    Coalesce (first non-empty input)
    
    Regex selection
    
    Sequence
    Sequence
    
    Count values
    
    Get value by index
    
    Sequence values to indexes
    
    Sort
    
    Substring
    Substring
    
    Strip postfix
    
    Strip prefix
    
    Strip URI prefix
    
    Substring
    
    Until character
    
    Template
    Template
    
    Evaluate template
    
    Tokenization
    Tokenization
    
    Camel case tokenizer
    
    Tokenize
    
    Uncategorized
    Uncategorized
    
    Convert currency values
    
    jq
    
    Validation
    Validation
    
    Validate date after
    
    Validate date range
    
    Validate number of values
    
    Validate numeric range
    
    Validate regex
    
    Value
    Value
    
    Constant
    
    Constant URI
    
    Dataset parameter
    
    Default Value
    
    Empty value
    
    Input hash
    
    Random number
    
    Read parameter
    
    ULID
    
    UUID
    
    UUID Convert
    
    UUID Version
    
    UUID1
    
    UUID1 to UUID6
    
    UUID3
    
    UUID4
    
    UUID5
    
    UUID6
    
    UUID7
    
    UUID8
- Cool IRIs
  Cool IRIs
- Lift Tabular Data
  such as CSV, XSLX and Database Tables
  Lift Tabular Data
- Lift Hierarchical Data
  such as JSON and XML files
  Lift Hierarchical Data
- Lift Web API Data
  Lift Web API Data
- Workflow Reconfiguration
  Workflow Reconfiguration
- Incremental Database Loading
  Incremental Database Loading
- Connect to Snowflake
  cloud datawarehouse via JDBC
  Connect to Snowflake
- Active learning
  Use AI/ML to learn linking rules
  Active learning
- Link Intrusion Detection Systems to Open-Source INTelligence
  such as STIX
  Link Intrusion Detection Systems to Open-Source INTelligence
  - Define the interfaces
    
    Define the interfaces
  - Define the need
    
    Define the need
  - lift data from STIX 2.1 data of mitre attack
    
    lift data from STIX 2.1 data of mitre attack
  - lift data from YAML data of hayabusa sigma
    
    lift data from YAML data of hayabusa sigma
  - link IDS event to KG
    
    link IDS event to KG
  - link IDS event to KG via cmem
    
    link IDS event to KG via cmem
- Project and Global Variables
  Project and Global Variables
- Evaluate Template Operator
  Evaluate Template Operator
- Build Knowledge Graphs from Kafka Topics
  Build Knowledge Graphs from Kafka Topics
Explore and Author
Explore and Author
- Workspace Selection and Configuration
  Workspace Selection and Configuration
- Graph Exploration
  Graph Exploration
  - Building a Customized User Interface
    
    Building a Customized User Interface
    
    Workflow Trigger
    
    Workflow Trigger
    
    Node Shapes
    
    Node Shapes
    
    Property Shapes
    
    Property Shapes
    
    Datatypes
    
    Datatypes
  - Statement Annotations
    
    Statement Annotations
  - Versioning of Graph Changes
    
    Versioning of Graph Changes
- Business Knowledge Editor Module
  Business Knowledge Editor Module
- Query Module
  Query Module
- Thesauri Management
  Thesauri Management
- Vocabulary Catalog
  Vocabulary Catalog
- Charts Catalog
  Charts Catalog
- Embedding Services via the Integrations Module
  Embedding Services via the Integrations Module
Consume
Consume
- Consuming Graphs in Power BI
  Consuming Graphs in Power BI
- Consuming Graphs in Redash
  Consuming Graphs in Redash
- Consuming Graphs with SQL Databases
  Consuming Graphs with SQL Databases
- Provide Data in any Format via a Custom API
  Provide Data in any Format via a Custom API
- Populate Data to Neo4j
  Populate Data to Neo4j
- Populate Data to Apache Kafka
  Populate Data to Apache Kafka
Deploy and Configure
Deploy and Configure
- System Architecture
  System Architecture
- Requirements
  Requirements
- Installation
  Installation
  - Scenario: Single Node Cloud Installation
    
    Scenario: Single Node Cloud Installation
  - Scenario: Local Installation
    
    Scenario: Local Installation
  - Migrating Stores
    
    Migrating Stores
- Configuration
  Configuration
  - Docker Orchestration
    
    Docker Orchestration
  - Build
    
    Build
    
    Plugin Reference
    
    Plugin Reference
    
    Activity Reference
    
    Activity Reference
  - Explore
    
    Explore
    
    Explore backend
    
    Explore backend
    
    General
    
    OAuth
    
    Store
    Store
    
    GraphDB
    
    HTTP
    
    In-Memory
    
    Neptune
    
    Virtuoso
    
    Graph resource pattern
    
    Graph resource pattern
  - Keycloak
    
    Keycloak
    
    Changing Passwords and Keys
    
    Changing Passwords and Keys
    
    Using an external Keycloak
    
    Using an external Keycloak
  - Quad-Store
    
    Quad-Store
  - Reverse Proxy
    
    Reverse Proxy
  - Access Conditions
    
    Access Conditions
  - Label Resolution and Full-Text Search
    
    Label Resolution and Full-Text Search
  - Production-Ready Settings
    
    Production-Ready Settings
  - Caveats
    
    Caveats
Automate
Automate
- cmemc
  Command Line Interface
  cmemc
  - Installation
    
    Installation
  - Configuration
    
    Configuration
    
    File-based
    Configuration
    
    File-based
    
    Environment-based
    Configuration
    
    Environment-based
    
    Completion Setup
    
    Completion Setup
    
    Getting Credentials
    from External Process
    
    Getting Credentials
    
    Certificate Handling
    and SSL verification
    
    Certificate Handling
  - Invocation
    
    Invocation
    
    Docker Image
    
    Docker Image
    
    Github Action
    
    Github Action
    
    Gitlab Pipeline
    
    Gitlab Pipeline
    
    SPARQL Scripts
    
    SPARQL Scripts
  - Workflow Execution
    and Orchestration
    
    Workflow Execution
  - Troubleshooting
    and Caveats
    
    Troubleshooting
  - Command Reference
    
    Command Reference
    
    Admin
    
    Admin
    
    Acl
    
    Acl
    
    Client
    
    Client
    
    Metrics
    
    Metrics
    
    Migration
    
    Migration
    
    Store
    
    Store
    
    User
    
    User
    
    Workspace
    
    Workspace
    
    Python
    
    Python
    
    Config
    
    Config
    
    Dataset
    
    Dataset
    
    Resource
    
    Resource
    
    Graph
    
    Graph
    
    Imports
    
    Imports
    
    Validation
    
    Validation
    
    Project
    
    Project
    
    Variable
    
    Variable
    
    Query
    
    Query
    
    Vocabulary
    
    Vocabulary
    
    Cache
    
    Cache
    
    Workflow
    
    Workflow
    
    Scheduler
    
    Scheduler
- Processing Data with variable input Workflows
  Processing Data with variable input Workflows
- Scheduling Workflows
  Scheduling Workflows
- Continuous Integration
  and Delivery
  Continuous Integration
Develop
Develop
- Accessing Graphs with Java Applications
  Accessing Graphs with Java Applications
- Python Plugins
  Python Plugins
  - Installation and Usage
    
    Installation and Usage
  - Development
    
    Development
  - Setup and Configuration
    
    Setup and Configuration
- cmempy - Python API
  cmempy - Python API
- cmemc - Python Scripts
  cmemc - Python Scripts
- Build (DataIntegration) APIs
  Build (DataIntegration) APIs
- Explore backend APIs
  Explore backend APIs
Release Notes
Release Notes
- Corporate Memory 25.2.1
  Corporate Memory 25.2.1
- Corporate Memory 25.1.2
  Corporate Memory 25.1.2
- Corporate Memory 24.3.2
  Corporate Memory 24.3.2
- Corporate Memory 24.2.1
  Corporate Memory 24.2.1
- Corporate Memory 24.1.3
  Corporate Memory 24.1.3
- Corporate Memory 23.3.2
  Corporate Memory 23.3.2
- Corporate Memory 23.2.1
  Corporate Memory 23.2.1
- Corporate Memory 23.1.3
  Corporate Memory 23.1.3
- Corporate Memory 22.2.3
  Corporate Memory 22.2.3
- Corporate Memory 22.1
  Corporate Memory 22.1
- Corporate Memory 21.11
  Corporate Memory 21.11
- Corporate Memory 21.06
  Corporate Memory 21.06
- Corporate Memory 21.04
  Corporate Memory 21.04
- Corporate Memory 21.02
  Corporate Memory 21.02
- Corporate Memory 20.12
  Corporate Memory 20.12
- Corporate Memory 20.10
  Corporate Memory 20.10
- Corporate Memory 20.06
  Corporate Memory 20.06
- Corporate Memory 20.03
  Corporate Memory 20.03
- Corporate Memory 19.10
  Corporate Memory 19.10
Tutorials
Tutorials

Table of contents

Parameter

TransformOperator

Remove duplicates¤

Removes duplicated values, making a value sequence distinct.

Parameter¤

None

2025-08-13 2025-08-12

Comments

Copyright © 2025 eccenca GmbH - CC-BY-SA-4.0 - Imprint

Made with Material for MkDocs

Cookie consent

We use cookies to recognize your repeated visits and preferences, as well as to measure the effectiveness of our documentation and whether users find what they're searching for. With your consent, you're helping us to make our documentation better.

Google Analytics
GitHub

Manage settings