Bringing data together is not always the easiest thing, especially if your data infrastructure is a hybrid of cloud and on-premise systems.
There are several tools available on the market that are used for traditional data transformation works. Sometimes, these are like using a sledgehammer to crack a nut. While supporting Wood for Trees (WfT), an analytics company who are experts in the charity sector, we came across an interesting challenge that was not unique to them.
WfT wanted to support their customers by bringing real-time operational data into a separate hosted analytics environment, enabling real-time data synchronisation for on-time analytics and decision making. But they wanted to manage this without having to pull any customer data, thus giving the customer control, and without the need for VPNs or other expensive technology.
The key areas of importance were:
- Security – the database is not open directly to a third party
- Complexity – the solution is lightweight and easy to install
- Cost – the cost is comparable to the complexity
MyLife Digital, the technical partners, looked at several products as potential solutions, but none of them appeared to have the necessary lightweight design or fit the other requirements set out by WfT and their customers. So, they designed and built their own solution, and this was a project that I was lucky enough to be involved in.
Step forward, DeltaFlux…
How does it work?
This lightweight solution consists of three key elements.
- A lightweight agent that:
- Detects and transmits changes in an SQL Server database to a cloud-based queue
- Regularly ‘dials home’ to confirm that it is up-and-running and healthy
- Obtains centralised configuration from a backend server for its activities
- Is easy to install
- A backend application comprising APIs and a UI for:
- Registering new client agents and associating them with a queue
- Viewing the health of deployed agents
- Viewing queue health
- A queue for each data source feed
With this in place, WfT are now able to pull their data in real-time from the individual queues and then push it into any desired customer database – both production and pre-production where analysis and data transformation can occur. All this is accomplished safely and securely. It’s transformed the way WfT offers analytics and insight for their customers.
The DeltaFlux project has been great fun to be involved with, and MyLife Digital aims for future releases to bring even better results.