Big data has turned out to be a key factor behind growing organizations scaling up their customer bases exponentially. Data already collected enables teams to automate key business operations, which pays for itself by bringing even more customers onboard without the small sizes of the teams becoming bottlenecks. This runaway chain reaction of big data is the reason some startups have grown from zero to millions of customers in just a couple of years. Indeed, big data may be the key to turning your startup into an industry leader. Let's study this evolving big data and its business impacts on three industries, and learn lessons that you can apply to your own company.
Big data started off as a term described through its technical characteristics. You have probably read elsewhere that big data is characterized by the three V's of data — volume, velocity, and variety — and the technologies to process them such as Hadoop.
In 2011, big data was still climbing up Gartner's hype cycle. Back then, big data and the processing software Hadoop were practically interchangeable terms. But as early as 2014, they were dropping into the trough of disillusionment and dropped off the hype cycle entirely by 2015.
At first glance, that looked bad for big data but there was an important caveat — Gartner correctly observed that big data had dropped off not because it had become irrelevant but because it was maturing into business as usual. The three V's of data still apply and software technologies remain key components of the big data ecosystem. But they're no longer its focus. Instead, the technology focus of big data has evolved into the business- and organization-focused philosophies of digital transformation and Industry 4.0 in every vertical.
This paradigm shift in focus becomes clear upon exploring big data and its business impacts on some verticals.
Fanatics, Inc. is a leader in selling sporting goods of major sports leagues and college teams. They grew sales from $250 million in 2010 to $3 billion in 2021. During this period, their market valuation went up from around $3 billion in 2013 to almost $13 billion in 2021. They now get 250 million visitors every year to the hundreds of online stores and brick-and-mortar shops that they own or manage on behalf of sports teams. From their technology partners like AWS and SingleStore, we have insights into how big data helped them scale up.
Fanatics started as a digital-native retailer focused on sports apparel. Over the years, they had already built up a big data setup composed of self-managed software like SQL Server for databases, Apache Hive for an SQL view on top of data stored in the Hadoop HDFS filesystem, and Lucene for full-text search. A retailer of this scale receives data that pushes the limits on all three axes — volume, velocity, and variety. The extreme variety had started causing functionality roadblocks. Since a relational database was the source of truth, all the data had to be transformed into relational models after data collection, but before storing. Other software like Lucene stored subsets of the data in their own storage silos and in their preferred formats.
All this made the setup unwieldy and brittle. If management wanted new features or use cases to be brought up quickly in preparation for a major sporting event, multiple systems and their data storage formats had to be reconfigured. Data scientists, managers, and executives were seeing different subsets of the data. Sometimes, important data was not available at all to managers and executives.
Such cases of the technology tail wagging the business dog due to software limitations were not only common in these legacy big data systems, but even tolerated as their inherent complexity.
Thankfully, big data has evolved toward placing business value and organizational needs on top. Fanatics, Inc. made many changes towards this evolution.
They started storing all data raw in their native formats on Amazon S3 data lakes that ensured that no information was dropped during data ETL transformations. Further, this migration enabled their business analysts, data scientists, and software engineers to use any arbitrary schema suitable to their particular task while reading the data, a concept known as schema-on-read. Without fixed schemas to hobble them, they could quickly implement any new features requested by executives.
They also moved the data to a common platform called SingleStore shared by all teams. This enabled the business intelligence teams to access all the data to generate latest real-time reports and data visualizations that their managers and executives needed for decision-making.
The overall scalability of the system was improved by moving to an event-driven architecture that used real-time stream processing systems like Apache Kafka. The system could now handle massive spikes in visitors that occurred during major sporting events. Executives could therefore make bold business decisions, envision new products, target new customers, and look for new opportunities in marketing, secure in the knowledge that their systems would not choke.
If you are running or planning an e-commerce venture, you too can get competitive advantages from big data and its business impacts on aspects like pricing optimization, demand forecasting, customer experience and service, omnichannel marketing including social media, product recommendations, semantic search, and data capture.
Every business requires legal services involving:
Legal-tech startups like Clerky, Ironclad, and Outlaw have attempted to provide this help conveniently through online services. But how do these small startups with barely 50 to 100 employees provide complicated legal services to hundreds of customers?
The answer lies in an aspect of big data that wasn’t usually associated with it in the past — intelligent automation. Earlier, big data analysis and intelligent automation were seen as separate fields because the former largely worked with structured data while the latter worked with unstructured data like images, text, and videos.
The skills required for each were also different — data engineering and business intelligence for big data analysis, machine learning and computer vision for intelligent automation. Whenever possible, companies resorted to business process outsourcing and robotic process automation to manually convert unstructured data to structured data because machine learning and computer vision engineers were few and the techniques used were not so capable.
But artificial intelligence and all its subfields — machine learning, deep learning, computer vision, and natural language processing — improved exponentially in their capabilities after 2012. They could understand images and text the way humans do. Businesses started using them and saw the benefits of such intelligent automation — increased profit, better operational efficiency, and reduced cost. They realized that by integrating them with their big data systems, they can uncover business insights that were previously locked up inside unstructured data like text, images, and videos.
Big data systems nowadays are tightly integrated with machine learning, deep learning, and other AI capabilities. Business intelligence teams can run machine learning models as easily as they run database queries. There are cloud services like Amazon SageMaker and Google AutoML that even enable point-and-click creation of new machine learning models without knowing anything at all about machine learning.
These small legal-tech startups are using such evolved big data systems to provide their services. Big data systems are not merely for business improvements but the very foundations of their core businesses. For example:
Choice Logistics is a logistics firm that adopted big data into their business workflows. In their digital transformation journey, they describe their vision of an evolved big data system to improve end-to-end operational efficiencies. They envision integrating not just their assets, employees, and Internet of Things (IoT) sensors but also those of their partners, customers, and transportation providers. With such a 360-degree end-to-end view of the entire supply chain, they can deploy better predictive analytics and schedule optimizations for cost reductions, increased profits, and better operational efficiency. They are aiming for greater velocity, efficiency, collaboration, mobility and accuracy.
Again, we see here that the focus is on the business and organization, not on big data technologies.
If you are planning a similar logistics startup, you should consider integrating warehouse automation and intelligent document processing to reduce manual processes and improve your operational efficiency further.
From these three case studies, we can draw some lessons about the evolving big data that apply to whichever vertical you’re targeting through your startup, be it healthcare, biotech, oil and gas, or something else:
There's a lot more to be said about big data and its role in your digital transformation. A good big data system requires your business executives, your domain experts, and our technical experts to team up.
Contact us with your needs and let's talk.
