Big Data, Big Rubbish?
- Senior Analytics Consultant
I am a result-driven and enthusiastic professional with extensive experience in Business, Financial, Statistical and Data Analysis, in diverse business domains and industries. …
VIEW PROFILE
“Answers and Tips to prevent your Big Data Deployments from becoming Big Rubbish”
Big Data has been trending in social media for years now. The concept is not a new one and it includes a set of different technologies that provide different applications for different industries. Leveraging from diverse Big Data sets that are available internally or externally, Big Data is characterized by the variety, the volume and the velocity that is produced and therefore, cannot be processed or analysed by more conventional methods such as Databases.
However, there is still a lot of confusion around the mix of technologies that are required, with productised solution platforms currently available in the market, still at their very early stage and with a total cost of ownership in order to gather, store, maintain and analyse Big Data still very high.
So, let’s agree on one thing; Big Data still remains a hype and a buzzword rather than a feasible solution option, especially for organisations that are not large enough to invest big on Big Data.
On the other side, and by all means, Big Data investments will certainly be proven to be beneficial in the near future, especially when combined with analytics capabilities and a clear vision for the future. This article provides answers to key questions and tips for creating a clear strategy to deliver Big Insights and prevent Big Data investment from becoming Big Rubbish.
What should a Big Data solution be?
As for most problems in today’s complicated world, there is not a single solution that fits all problems. You will need to identify the optimal mix of technologies and create a scalable, flexible environment that will serve your needs and plans for your future and align your business objectives and strategic priorities.
But, as Big Data will continue to grow and become even Bigger, Advanced Analytics always needs to be the core component in any Big Data solution in order to effectively operationalise and optimise data related processes and derive actionable insights and business value.
Big Data without Advanced Analytics capabilities will simply result in data overload, noise and unavoidably Big Rubbish. It is not the increasing volume of Big Data that will answer your business problems or the increasing computational power or storage capacity, but the analytical algorithms that will allow you to mine Big Data sets and discover hidden patterns and associations.
What type of Big Data do I need?
Again, there is not a single golden rule that should always be followed, as the answer relies on the industry, the activities and the environment in which each organisation operates, and most importantly on the business problem that needs to be addressed.
Big Data sources vary in terms of structure, origin and nature. For example, for a financial institution that aims to improve customer service, Big Data can originate from structured CRM/Marketing, Sales and Transactional data sets, unstructured Social Media text and external Social Economic data from surveys. For a manufacturing company that wants to reduce production downtime, Big Data can originate from Asset Management Systems, Supervisory control and data acquisition (SCADA) systems, Inspections and Production yields, and also Weather Data and Internet of Things (IoT) Sensors readings from machines’ equipment.
How Big should my Big Data be?
Analysing Big Data sources can result in Big Insights, however they can also result in Big Noise and Big Rubbish. A balance between Big Data sources and targeted, representative samples (Small Data), and also between traditional modelling techniques and artificial intelligence, deep learning and machine learning algorithms, should be maintained. Variable selection and sampling is still important for an accurate statistical model and access to Big Data is the key component for that. It will provide data scientists the required bandwidth to build differentiated analytical models that can address different business problems and reveal associations and correlations, while Small Data, or appropriate relevant data samples will help you to reveal the root causes and produce accurate predictions and targeted actions for future events.
What other things should I consider?
Even if Big Data is technology-intensive deployments, peoples’ skills are still and will remain important for deriving business value from data processes. They will prevent your Big Data efforts becoming Big Rubbish and will increase return on investment. Experienced data scientists with appropriate skills and training, combined with a project governance system will definitely add clarity to your efforts and create a competitive advantage for the future.
More Analytics Blog Posts
Read more >>
Read more >>
- SPSS Statistics Version 26 includes new statistical tests, enhancements to existing statistics and scripting procedures…Read more >>
- Top tips and tricks for running ANOVA and non-parametric tests along with more advanced forms…Read more >>
- SPSS Statistics Version 26 includes new statistical tests, enhancements to existing statistics and scripting procedures…Read more >>
- Healthcare systems face multiple challenges, including ageing populations, evolving healthcare needs and rising costs. Tailored…Read more >>
- Forecasting predicts future values of a particular quantity based on previously observed values of that…Read more >>
- Alan shows you how identify new, data driven, re-order points for your stock items that…Read more >>
- In this short video you will learn how to run a Time Series model within…Read more >>
- Alan now looks more closely at the quality of information you hold on your inventory.…Read more >>
- Alan takes you through the Smart Inventory Management interactive dashboard, showcasing an overview of your…Read more >>
- In this third video, Alan shows you how to extract the key findings from a…Read more >>
- Join us at the Technology Event of the Year! Watch Rob McCullagh present Presidion’s innovative…Read more >>
- This second video concentrates on using Hold-out samples to create more robust predictive models. In…Read more >>
- SPSS is a fully customisable analytical software. Due to the highly flexible nature of SPSS…Read more >>
- In this video, the first of a series, Alan takes you through running a Decision…Read more >>
- If you have ever found that your company has engaged with Analytics initiatives that have…Read more >>
- In the 4th video of our series you will learn how to build stacked tables,…Read more >>
- In this video you will learn how to tailor your tables, run T-test & create…Read more >>
- In June 2018, I was invited to speak at the UK Manufacturing and Supply Chain…Read more >>
- In this video you will learn how to protect respondent's identity and boost the statistical…Read more >>
- Best practices and key considerations before, during and after deploying an analytical initiative to maximise…Read more >>
- In the first video of the IBM SPSS Custom Tables series you will learn: How…Read more >>
- I’ve often seen analytics described primarily by way of the technical complexity of the techniques.…Read more >>
- Being able to effectively measure a RoI on the back of an Advanced Analytics project…Read more >>
- Throughout industries analytics has taken on an almost ubiquitous nature of late. There are increasing…Read more >>
- Analytics is being discussed more and more often in top table conversations, but why do…Read more >>
- Watch Pierre Baviera speaking at “Demystifying Data” Business Breakfast at the Imperial Hotel (Cork), in…Read more >>
- A 3 step process to Understand, Assess and Audit your Data for Advanced Analytics.Read more >>
- Presidion’s Framework Approach for a Technology Review involves a 3 step process, leading to the…Read more >>
- With inspirational quotes from Data and Analytics gurus worldwide.Read more >>
- Take the Analytical Maturity model presented in Thomas Davenport’s “Competing on Analytics”. Would you consider…Read more >>
- There are times when deploying advanced analytics initiatives can feel like you are in a…Read more >>
- After our last A4B “War Stories - Data Analyst Down”, we asked the keynote speaker,…Read more >>
- The truth is that there are many factors and forces at work that help build…Read more >>
- While I’m not sure this statement is true when referring to people, I really believe…Read more >>
- As I observe and read about the raging fires causing massive destruction, I have been…Read more >>
- After our last A4B we asked the keynote speaker, Anthony O’Neill (Director, Analytics Centre of…Read more >>
- The science and psychology to get the 'Bikini Body' you have always wanted (with Predictive…Read more >>
- The reasons behind, and some hints to help you during difficult conversations and challenging situations!Read more >>
- When you start with Analytics you might feel overwhelmed, you have all that incredible power…Read more >>
- Preventative and Conditional Maintenance are often mistaken with Predictive Maintenance. This animation gives you an…Read more >>
- Reports from the insurance industry consistently highlight that the quality of customer experience remains the…Read more >>
- Unlike the consumer sector, manufacturers have been using IoT for years. Properly focused IoT capabilities…Read more >>
- The World is going through the Digital Revolution. What does it practically mean? Well, it…Read more >>
- Does your organisation struggle with understanding how to use analytics? These days it is not…Read more >>
- As a CEO, it’s important that you not just have some good one liners to…Read more >>
- Predictive Maintenance, the modern approach to maintenance that maximises productivity while reducing overall costs.Read more >>
- It is a truism that business need to win new customers and retain existing customers.…Read more >>
- Oil and Gas companies are borrowing from data rich industries to transform their maintenance procedures.…Read more >>
- Will the reign of terror ever end? Can we fight ISIS? Here are the tangible…Read more >>
- Answers to key questions and tips for creating a clear strategy to deliver Big Insights…Read more >>
- More often than not predictive analytics projects represent unfulfilled potential. Is your next project doomed…Read more >>
- Using asset specific data to predict where, when and why a specific asset will fail…Read more >>
- Can one of the most common applications in Retail sector (Market Basket Analysis), determine Next…Read more >>
- With the explosion of Big Data in the past few years there has been a…Read more >>
- One of the most important elements of any data analytics project within any business is…Read more >>
- How many times do you have to be told something before you believe it? Do…Read more >>
- Do you think you are a “good” Analyst? Are there certain traits, attributes and qualities…Read more >>
- Good predictive models take the time and experience of skilled analysts to develop. How do…Read more >>
- We hear that “Data is the new Oil” but how do you extract real value…Read more >>
- Fraud in organizations can be epidemic if not curbed at the inception. According to a…Read more >>
- Were you ever faced with identifying insights from open ended text data? From call center…Read more >>
- The Irish playwright was on the money - no matter what way you look at…Read more >>
- There are two certainties in life, death and taxes. This proverb rings true especially for…Read more >>