In this section
Data Mesh — federated data streaming between platform tenants
The problem this solves
Multi-tenant platforms create data silos by design. Each team runs its own services, its own databases, and its own pipelines — and when one team needs data from another, the answer is usually a one-off API integration, a manual export script, or a shared database that turns into a coupling nightmare.
Building and maintaining bespoke ETL pipelines for every cross-team data flow doesn't scale, is hard to audit, and breaks silently when schemas change.
SmoothGlue Data Mesh gives teams a platform-native, self-service way to publish and subscribe to data streams — without requiring a dedicated data engineering team or custom infrastructure.
What Data Mesh is
Data Mesh is SmoothGlue's federated data streaming layer. It lets platform tenants publish structured data from their services and subscribe to data published by other tenants — without direct service coupling or shared databases.
The model is producer/consumer: a team declares what data they publish (an origin), and other teams declare what they want to receive (a destination). The platform handles the transport, transformation, and delivery.
This is not a data warehouse or a batch ETL tool. Data Mesh is designed for continuous, event-driven data flows between live services running on the platform.
How it works
Publication and subscription model
Tenants interact with Data Mesh through three Kubernetes custom resources:
xdatamesh— the top-level resource that defines a data mesh instancexpublication— declares a data origin: what data is being published and from wherexsubscription— declares a data destination: who receives what, and how it is transformed
This declarative model means data flows are version-controlled, auditable, and self-documenting. Adding a new consumer does not require changes to the producer.
Transport layer
Under the hood, all data flows through NATS JetStream — a high-throughput, persistent messaging system built for distributed environments. JetStream provides at-least-once delivery guarantees, durable streams that survive consumer restarts, and backpressure handling for slow consumers. This means data is not lost if a destination service is temporarily unavailable.
Processing engine
The datamesh-core service processes all data in motion. It is a Go-based engine forked from Benthos, purpose-built for the platform's reliability and security requirements. Transformations between origin and destination are expressed in Bloblang — a domain-specific language for field mapping, filtering, and restructuring structured data — keeping transformation logic portable and readable without custom code.
Configuration UI
Platform operators and tenant teams configure data flows through the SmoothGlue Console using guided setup wizards: a 5-step flow for defining an origin and a 6-step flow for configuring a destination. The Console generates the underlying Kubernetes resources automatically.
The backend is a Django REST API; the frontend is a React application embedded in the SmoothGlue Console.
What's pre-integrated
| Capability | What SmoothGlue provides |
|---|---|
| 30+ input connectors | Databases, S3-compatible object storage, Kafka topics, HTTP endpoints, and more — available out of the box as origin sources or destination sinks |
| Bloblang transforms | Built-in transformation language; no custom code or separate transformation service required |
| Console wizard | Guided UI for origin and destination setup — no direct Kubernetes resource authoring needed for standard flows |
| NATS JetStream | Pre-deployed, pre-configured; persistent streams and at-least-once delivery are on by default |
| Kubernetes integration | XRDs are registered at platform install; data mesh resources are first-class platform citizens |
What you still own
- Your data schemas and how they evolve over time
- Bloblang transformation logic specific to your use case — the platform runs it, you write it
- Retention policies for your published streams
- Access decisions: which tenants are permitted to subscribe to your publications
- Source and destination credentials (stored in Vault, but created and managed by your team)