DeltaFlux – The Light Touch On Real-Time Data Synchronisation

The world of analytics has grown in complexity over the last few decades. The often-touted ‘cloud’ is supposedly where everyone should be, but the reality is that each business is on its own unique data journey. When companies are supporting analytics and business-intelligence initiatives, they are essentially trying to bring data together – be that in a data warehouse, a data lake, or some other new terminology.

Share This Post

Share on facebook
Share on linkedin
Share on twitter
Share on email

Bringing data together is not always the easiest thing, especially if your data infrastructure is a hybrid of cloud and on-premise systems. 

The problem 

There are several tools available on the market that are used for traditional data transformation works. Sometimes, these are like using a sledgehammer to crack a nut. While supporting Wood for Trees (WfT), an analytics company who are experts in the charity sector, we came across an interesting challenge that was not unique to them.  

WfT wanted to support their customers by bringing real-time operational data into a separate hosted analytics environment, enabling real-time data synchronisation for on-time analytics and decision making. But they wanted to manage this without having to pull any customer data, thus giving the customer control, and without the need for VPNs or other expensive technology.  

The key areas of importance were: 

  1. Security – the database is not open directly to a third party 
  1. Complexity – the solution is lightweight and easy to install 
  1. Cost – the cost is comparable to the complexity 

MyLife Digital, the technical partners, looked at several products as potential solutions, but none of them appeared to have the necessary lightweight design or fit the other requirements set out by WfT and their customers. So, they designed and built their own solution, and this was a project that I was lucky enough to be involved in. 

Step forward, DeltaFlux… 

How does it work? 

This lightweight solution consists of three key elements. 

  1. A lightweight agent that: 
  • Detects and transmits changes in an SQL Server database to a cloud-based queue 
  • Regularly ‘dials home’ to confirm that it is up-and-running and healthy 
  • Obtains centralised configuration from a backend server for its activities 
  • Is easy to install 
  1. A backend application comprising APIs and a UI for: 
  • Registering new client agents and associating them with a queue 
  • Viewing the health of deployed agents 
  • Viewing queue health 
  1. A queue for each data source feed 

With this in place, WfT are now able to pull their data in real-time from the individual queues and then push it into any desired customer database – both production and pre-production where analysis and data transformation can occur. All this is accomplished safely and securely. It’s transformed the way WfT offers analytics and insight for their customers.  

The DeltaFlux project has been great fun to be involved with, and MyLife Digital aims for future releases to bring even better results.   

Mark Lugg

Mark Lugg

Subscribe To Our Newsletter

Get updates and learn from the best

More To Explore

Blog

How the Development Team Organises their work

The Development Team follows the principles of “Agile Scrum”.

e.g. a daily Stand-up meeting to organise the day’s work and to update the Product Team, plus Planning and Retrospective “ceremonies”.

Opinion

Imagine

Fast forward x months, you have to forgive the inner mathematician, the lockdown has ended and the health impacts of the Coronavirus are behind us; people now move without fear of other humans; people get as much toilet paper and pasta as they want, delivered when they want; pensioners have returned to shops and spend time talking to checkout assistants while impatient shoppers in the queue behind them huff; non-essential workers are returning to ‘non-essential’ offices to do ‘non-essential’ jobs. The world can return to Business-As-Usual, but will it?

Find out how you can do more with data

Get in touch