Petabyte Archiving

Petabyte-Scale Archiving Infrastructure Technical Architecture of Documenos Archive Management

1. Scalable Infrastructure:

• Development based on ASP.NET Core
• PostgreSQL (v15 and later) database compatibility
• Architecture suited to horizontal and vertical scaling
• Cluster support
• Seamless access behind a load balancer
• Every server can connect to the same database.
• Capacity can be expanded horizontally by adding servers.

2. Archive Tiering and Data Flow
With background services:

• Automatic upload from the file system to the archive
• Uploading data from FTP to the archive
• Transfer between archive tiers
• Bulk extraction operations
• Physical purging of deleted data
• Services can be controlled per server.

3. Content Processing Services
For archived content:

• Image processing (thumbnails, resizing, EXIF/IPTC extraction)
• PDF text indexing
• OCR tasks
• Speech-to-text conversion in video files (100+ languages)
• Speech-to-text conversion in audio files (100+ languages)
• The generated texts are stored in the search indexes.

4. Data Model and Metadata Capacity
Support for 1.4 billion tables in dynamic datasets
For each table in dynamic datasets:

• Recommended use: 250 fields
• Technical upper limit: 1600 fields
• Storage capacity of ~4.2 billion rows of data

Datasets:

• Authorization-based visibility
• Excel upload / download
• Automatic data updates via event triggers

5. Security and Access Control

• LDAP integration
• Built-in password system
• Multi-factor authentication (OTP, email, SMS)
• IP-based access restrictions
• Group-based authorization
• Declaration that no backdoor is defined in the software