Technical choices

This project was designed to migrate data from one EDM system to another while preserving the associated metadata.

Find out what this service is for in this article.

Selected technologies

Usage

We chose a solution based on robust open source software: Apache Airflow¹ and Apache Kafka². We wanted this software to be simple and efficient to configure according to each need, so that everyone gains in productivity. Our solution can interrupt the process without losing what has already been processed, and can adjust its time/load ratio so as not to impact a production server while working in the background.

How the software is built

Extraction

With Airflow¹, a tool widely used in the AI world, we track the progress of the data migration. To guarantee stable processing, we use a JSON[^Json] pivot file that describes the document to be migrated. This pivot file brings together the document’s source, its metadata and the desired destination path. The pivot format is described with JSON Schema, so that it is independent of both target and source. Our solution can adapt to any data source, whether a REST API, a WebDAV interface, a file server or even scraping a website, etc.

Transformation

We extract the metadata from this source, and then:

• a mapping rule must be defined to our source data model,
• a rule is needed to define the destination path of this new data.

With this information, we write code that generates a JSON file (which we call the pivot file). It describes the object as a whole, its source and its destination metadata. This pivot-format file is then sent to our injectors, and the injection is automatic.

Load

Injection goes through a Kafka² queue, which receives the information and injects it into the target server according to the client’s requirements (impact on production, available server resources, one-off data migration or continuous injection, etc.).

Application to Alfresco

On Alfresco specifically, metadata from an unmaintained server is converted to follow the best practices of an Alfresco folder structure. This solution preserves the file’s creation date, whereas other methods copy the information and record the creation date as the time of the copy.

See our use case Use case: Pristy Connect data migration between two Alfresco instances to find out more.

Learn more

• Our data migration service
• Use case: Pristy Connect data migration between two Alfresco instances

ETL

the generic name for data migration tools. It stands for Extract-Transform-Load.

JSON Schema

While JSON is probably the most popular format for exchanging data, JSON Schema is the vocabulary that enables JSON data consistency, validity, and interoperability at scale.[^Json]

Airflow¹

an open source workflow management platform ³

Kafka²

an open source project under the Apache License 2.0: “stream-processing software for collecting, processing, storing and analysing data at scale”. ⁴