Connectors

A Connector enables Publisher to interface with various external storage systems. They provide standardized access to files and tables while managing authentication, monitoring, and data access functionality.

Required Connector Functionality

A Connector must fulfill these requirements:

Configuration

• Provide an HTML user interface for configuring external connections

  • Store encrypted credentials for external systems (e.g., service accounts or dedicated user accounts)

  • Generate an audit trail for configuration changes

Data Access

• Respond to requests about attached data by:

  • Listing files with current metadata (size, checksums, create/update dates)

  • Generating time-limited signed URLs for file retrieval

  • Listing tables with current metadata (column names & types, row count)

  • Dumping table data with optional column and row subsetting

  • Generating an audit trail for data access

Monitoring

• Monitor external systems for data changes

  • Notify Publisher of changes

• Log all external system access, including purpose and success/fail status

Connector Types

1. AWS S3 Storage

2. Google Cloud Storage

3. Microsoft Azure Blob Storage

4. PostgreSQL

5. HTTPS

While Publisher includes DNAstack-developed connectors, third-party developers can create additional connectors for other data sources. For information on developing custom connectors, refer to the Data Source Connector Developer Guide.