Organisations constantly produce information and data at a rate like never before. This data can unlock secrets in your business which in turn can help you make intelligent decisions.

But how?

Previously, it’s been very difficult because of the availability of compute and storage constraints. Big Data is a large volume of structured and unstructured data that can be analysed for insights. We use an analogy to help describe Big Data and the process of data gathering to clients.

We call it the ‘Data Spider’.

The Data Spider

A spider weaving a web will cement web anchor points with piriform silk to a supporting structure. Soft, resilient wet silk makes up the rest of the orb web which is used to capture insects that become trapped.

Similarly, a data spider orchestrates the ingestion of data from multiple anchor points. These supporting structures are your key business systems, databases, information exports, information endpoints and data silos. The anchor points form the foundation of the web which the data spider will use to traverse available data sources. Information is aggregated together to build an elastic mesh of data available for querying and consumption by business intelligence tools.

Simply put – a ‘Data Spider’, is an orchestration process powering the data gathering process for all the data in your business.

Once the data is gathered, it can be transformed and manipulated to meet the requirements for inclusion in a data warehouse or data lake. The transformation process moves data into a cohesive environment with defined relationships enabling correlation of data and the insights that businesses are looking for.

Depending on what the business is trying to achieve and the availability of data, real-time or delayed data can be aggregated together to form business intelligence and analytics.

Visual dashboards, automated reports and even automatic processes can be executed based on the data gathered and processed. AWS has a suite of services to make answering the hard questions, easier.

Do you have questions that are hard to answer? Idea 11 has designed data spiders for organisations to help reduce cost, provide insight into strategic business decisions and enable transparency for stakeholders around complex processes.

Interested in what an AWS-based data spider can mean for your business? Get in touch.