Architecting a data platform on public cloud for a large enterprise

Is your organisation composed of different business areas, each of whom have their own independent aspirations in terms of landing data, curating data, building analytics models and running data-intensive applications?

On-premises vs public cloud

With an organisation of this size, there are often two philosophies: one which tries to centralise all things data and one which allows business areas to operate with autonomy. This is the centralisation vs federation debate, which is a concept we dance with on-premises.

With federation, you empower your organisation but risk the wild west where businesses may ingest the same data twice and where businesses are adopting conflicting data models, naming conventions, metadata standards, leading to a potentially divergent and inconsistent data landscape.

With centralisation, you see a single centre of excellence carry out the data work on behalf of your business areas to a consistent enterprise standard but often at the risk of creating a bottleneck that impedes the entire organisation to move at pace with data.

However, on public cloud, there are designs—and not just patterns — that can bridge these two positions, achieving the best of both.

On-premises there are also clear delineations between enterprise systems; classified into systems of record (SoR), systems of insight (SoI) and systems of engagement (SoE). An…