Set up a Data Mesh origin
What you'll build
A live data stream published from a source system into the platform's Data Mesh. Other teams can subscribe to this stream without direct database access — no credentials shared, no direct connections opened.
Before you begin
- SmoothGlue platform running and healthy (
smoothglue status) - Console access with OrgPrivileged role or higher
- Credentials for your source system (hostname, username, password or IAM role)
Step 1 — Navigate to Data Mesh
In the Console left sidebar, select Data Mesh → Origins → New Origin.
The origin wizard opens.
Step 2 — Choose origin type
Select the connector type that matches your source system from the dropdown.
Common options:
| Connector | Use for |
|---|---|
| PostgreSQL | Relational tables, supports CDC |
| MySQL | Relational tables, supports CDC |
| S3 | Object storage, batch polling |
| Kafka | Existing Kafka topics |
| HTTP Endpoint | REST or webhook sources |
| Redis | Key-value and pub/sub streams |
The wizard shows only the fields relevant to the selected type. Select your connector and click Next.
Step 3 — Configure connection
Enter your source system's connection details. Fields vary by connector type.
PostgreSQL example:
| Field | What to enter |
|---|---|
| Host | Database hostname or IP address |
| Port | 5432 (default) |
| Database | Database name |
| Username | Read-only service account username |
| Password | Stored as a Kubernetes secret — not visible after save |
| SSL Mode | require (recommended for production) |
The Console validates the connection before allowing you to proceed. If validation fails, check that:
- The host is reachable from the cluster network
- The service account has
SELECTprivileges on the target tables - SSL mode matches the server's requirements
Fix any connection errors before clicking Next.
Step 4 — Define the publication
Configure what data to stream and how.
| Field | What to enter |
|---|---|
| Publication name | A slug identifying this stream (e.g. orders-stream) |
| Tables / Query | Select tables from the picker, or write a SQL query (connector-dependent) |
| Polling interval | How often to check for new rows (e.g. 30s). Not used when CDC is enabled. |
| Change-data-capture (CDC) | Enable if your connector supports it — streams row-level changes in near real time instead of polling |
Click Next.
Step 5 — Review and publish
The Console shows a summary before creating the stream:
- Origin type — the connector you selected
- Connection — host and database name shown; password redacted
- Publication name — the slug you entered
- Tables / Query — what data will be streamed
- Delivery mode — polling interval or CDC
Review all fields. Click Publish.
Expected output in the Console:
Origin created: orders-stream
Status: Provisioning...
✓ xpublication resource created
✓ datamesh-core started
Status: Active
Status changes to Active within 60 seconds.
Verify
Confirm the origin is running:
- Console → Data Mesh → Origins → select your origin
- Check Status: Active
- Confirm the Messages/sec counter is incrementing
Copy the Publication ID from the origin detail page. Subscribers need this ID to connect.
What happened under the hood
The Console created an xpublication custom resource in the datamesh namespace. The datamesh-core engine (a Go-based Benthos fork) is now polling or streaming your source and publishing messages to a NATS JetStream subject. Subscribers can connect to that subject without any direct access to your source database.