Big Data and the Powerful Business Impacts It Has in 2021

Karthik Shiraly
May 5, 2022
Big data and its business impacts: man stacking coins with a graph of growth over it

Big data has turned out to be a key factor behind growing organizations scaling up their customer bases exponentially. Data already collected enables teams to automate key business operations, which pays for itself by bringing even more customers onboard without the small sizes of the teams becoming bottlenecks. This runaway chain reaction of big data is the reason some startups have grown from zero to millions of customers in just a couple of years. Indeed, big data may be the key to turning your startup into an industry leader. Let's study this evolving big data and its business impacts on three industries, and learn lessons that you can apply to your own company.

What Does Big Data Mean Today?

Big data started off as a term described through its technical characteristics. You have probably read elsewhere that big data is characterized by the three V's of data — volume, velocity, and variety — and the technologies to process them such as Hadoop.

In 2011, big data was still climbing up Gartner's hype cycle. Back then, big data and the processing software Hadoop were practically interchangeable terms. But as early as 2014, they were dropping into the trough of disillusionment and dropped off the hype cycle entirely by 2015.

At first glance, that looked bad for big data but there was an important caveat — Gartner correctly observed that big data had dropped off not because it had become irrelevant but because it was maturing into business as usual. The three V's of data still apply and software technologies remain key components of the big data ecosystem. But they're no longer its focus. Instead, the technology focus of big data has evolved into the business- and organization-focused philosophies of digital transformation and Industry 4.0 in every vertical. 

This paradigm shift in focus becomes clear upon exploring big data and its business impacts on some verticals.     

How Big Data Helped an E-Commerce Company Grow Sales by 12x and Raise Market Valuation by 4x

Big data and its business impacts: miniature, full shopping cart on top of a laptop with graphs on the screen

Fanatics, Inc. is a leader in selling sporting goods of major sports leagues and college teams. They grew sales from $250 million in 2010 to $3 billion in 2021. During this period, their market valuation went up from around $3 billion in 2013 to almost $13 billion in 2021. They now get 250 million visitors every year to the hundreds of online stores and brick-and-mortar shops that they own or manage on behalf of sports teams. From their technology partners like AWS and SingleStore, we have insights into how big data helped them scale up.

Fanatics started as a digital-native retailer focused on sports apparel. Over the years, they had already built up a big data setup composed of self-managed software like SQL Server for databases, Apache Hive for an SQL view on top of data stored in the Hadoop HDFS filesystem, and Lucene for full-text search. A retailer of this scale receives data that pushes the limits on all three axes — volume, velocity, and variety. The extreme variety had started causing functionality roadblocks. Since a relational database was the source of truth, all the data had to be transformed into relational models after data collection, but before storing. Other software like Lucene stored subsets of the data in their own storage silos and in their preferred formats.

All this made the setup unwieldy and brittle. If management wanted new features or use cases to be brought up quickly in preparation for a major sporting event, multiple systems and their data storage formats had to be reconfigured. Data scientists, managers, and executives were seeing different subsets of the data. Sometimes, important data was not available at all to managers and executives.

Such cases of the technology tail wagging the business dog due to software limitations were not only common in these legacy big data systems, but even tolerated as their inherent complexity.

Placing the Business on Top

Thankfully, big data has evolved toward placing business value and organizational needs on top. Fanatics, Inc. made many changes towards this evolution.

They started storing all data raw in their native formats on Amazon S3 data lakes that ensured that no information was dropped during data ETL transformations. Further, this migration enabled their business analysts, data scientists, and software engineers to use any arbitrary schema suitable to their particular task while reading the data, a concept known as schema-on-read. Without fixed schemas to hobble them, they could quickly implement any new features requested by executives.

They also moved the data to a common platform called SingleStore shared by all teams. This enabled the business intelligence teams to access all the data to generate latest real-time reports and data visualizations that their managers and executives needed for decision-making.

The overall scalability of the system was improved by moving to an event-driven architecture that used real-time stream processing systems like Apache Kafka. The system could now handle massive spikes in visitors that occurred during major sporting events. Executives could therefore make bold business decisions, envision new products, target new customers, and look for new opportunities in marketing, secure in the knowledge that their systems would not choke.

If you are running or planning an e-commerce venture, you too can get competitive advantages from big data and its business impacts on aspects like pricing optimization, demand forecasting, customer experience and service, omnichannel marketing including social media, product recommendations, semantic search, and data capture.

How Big Data Helps Small Legal-Tech Startups Attract Hundreds of Customers

Big data and its business impacts: three lawyers looking at a tablet

Every business requires legal services involving:

  • Expensive lawyers
  • Plenty of complicated text inside contracts, agreements, and other legal documents     
  • Much time spent to read, understand, and write them     
  • Formalities such as signatures, court filings, and regulatory filings

Legal-tech startups like Clerky, Ironclad, and Outlaw have attempted to provide this help conveniently through online services. But how do these small startups with barely 50 to 100 employees provide complicated legal services to hundreds of customers?

The answer lies in an aspect of big data that wasn’t usually associated with it in the past — intelligent automation. Earlier, big data analysis and intelligent automation were seen as separate fields because the former largely worked with structured data while the latter worked with unstructured data like images, text, and videos. 

The skills required for each were also different — data engineering and business intelligence for big data analysis, machine learning and computer vision for intelligent automation. Whenever possible, companies resorted to business process outsourcing and robotic process automation to manually convert unstructured data to structured data because machine learning and computer vision engineers were few and the techniques used were not so capable.

But artificial intelligence and all its subfields — machine learning, deep learning, computer vision, and natural language processing — improved exponentially in their capabilities after 2012. They could understand images and text the way humans do. Businesses started using them and saw the benefits of such intelligent automation — increased profit, better operational efficiency, and reduced cost. They realized that by integrating them with their big data systems, they can uncover business insights that were previously locked up inside unstructured data like text, images, and videos.

Big data systems nowadays are tightly integrated with machine learning, deep learning, and other AI capabilities. Business intelligence teams can run machine learning models as easily as they run database queries. There are cloud services like Amazon SageMaker and Google AutoML that even enable point-and-click creation of new machine learning models without knowing anything at all about machine learning.

These small legal-tech startups are using such evolved big data systems to provide their services. Big data systems are not merely for business improvements but the very foundations of their core businesses. For example:

The Digital Transformation of a Logistics Firm Using Big Data

Worker wearing a hard hat walking around a warehouse

Choice Logistics is a logistics firm that adopted big data into their business workflows. In their digital transformation journey, they describe their vision of an evolved big data system to improve end-to-end operational efficiencies. They envision integrating not just their assets, employees, and Internet of Things (IoT) sensors but also those of their partners, customers, and transportation providers. With such a 360-degree end-to-end view of the entire supply chain, they can deploy better predictive analytics and schedule optimizations for cost reductions, increased profits, and better operational efficiency. They are aiming for greater velocity, efficiency, collaboration, mobility and accuracy.

Again, we see here that the focus is on the business and organization, not on big data technologies.

If you are planning a similar logistics startup, you should consider integrating warehouse automation and intelligent document processing to reduce manual processes and improve your operational efficiency further.     

What Conclusions Can We Draw?

From these three case studies, we can draw some lessons about the evolving big data that apply to whichever vertical you’re targeting through your startup, be it healthcare, biotech, oil and gas, or something else:

  • Small companies require big data too: The use of big data is not just for big companies. As we saw with Fanatics and the legal-tech startups, big data can be critical for growth. Without big data, your company may remain a small company forever.
  • Business and organization should shape your big data system: Your business and organizational needs should have priority over your big data system, not technologies. If it can't do something your business needs, look for alternatives.     
  • Big data requires business buy-in: Big data once assumed technologies are the critical factor and that if the right tools are used and the right data is collected, business insights would follow from big data analytics. In reality, management and organizational buy-in is crucial because, without it, these insights are not actionable.
  • Big data is not just about the volume of data: Don't assume that “big” only refers to large amounts of data. Big applies to the other axes too — velocity and variety. Even if your company processes small volumes from thousands of sources or handles thousands of different data formats, you still need big data. If your business involves real-time processing from mass data sources such as smartphones and IoT devices, you need a big data system.
  • Big data includes artificial intelligence, automation, data science, machine learning, and deep learning: These are all essential components of a modern big data system, enabling you to unlock business insights hidden in unstructured data. Often, machine learning models can be integrated right into a larger data pipeline. The big data system takes care of executing them on the most suitable hardware — GPUs for unstructured data and CPUs for structured data.     
  • Big data has moved off the shelf into the cloud: Hadoop became popular because it enabled distributed data processing at scale on inexpensive commercial off-the-shelf (COTS) hardware. But cost savings on the hardware were lost in all the engineering complexities it brought with it, requiring expensive talent to be hired. Modern big data systems can be easily run on public cloud computing platforms like AWS without such complexity or expensive talent.     
  • Big data solutions come as data lakes and data warehouses: The industry realized the need to tightly integrate big data software, storage, data management, deployment, security, and governance. Without that, the big data system becomes complex and lacks the agility to respond to business needs quickly. These integrated packages are available to you in the form of data lakes and data warehouses.

Talk to Us About Your Digital Transformation

There's a lot more to be said about big data and its role in your digital transformation. A good big data system requires your business executives, your domain experts, and our technical experts to team up.

Contact us with your needs and let's talk.