Automating the Future: Data Trends Powering the AI Revolution

TLDR

Snowflake Inc., a cloud data platform company, analyzed usage data from its customer base to identify trends in how organizations leverage data and AI. The goal was to provide insights into how leading companies use data to drive innovation and results.

Methodology: The analysis examines data usage from ~7,800 Snowflake customers over a 12-month period ending Jan. 31, 2023. It identifies patterns in how these organizations interact with data to power improvements.

Key findings:

• Companies are connecting data globally across different cloud platforms, with cross-cloud growth of 207% to enable collaboration and resilience.
• Organizations want to bring more workloads like AI to their data, rather than moving data to the work. The number of jobs run in the Snowflake platform grew 64%, indicating usage beyond just storage.
• Governance policies surged as data regulations increased, with masking policies up 205% and object tagging up 40x. This shows compliance efforts.
• Usage of automated warehouse resizing rose 71%, demonstrating a preference for leveraging automation to optimize costs and resources.

Recommendations:

• Modernize data infrastructure to remove barriers to collaboration and allow teams to securely access live, shared data.
• Support multiple programming languages like Python and SQL against the same data copy to avoid new silos.
• Refocus governance to safely enable broader data usage, not just limit access. Apply one consistent governance model.
• Embrace automation like cloud resource optimization to minimize costs and free up teams from maintenance.

Thinking Critically

Implications

• If organizations connect data across systems and geographies, it could enable faster insights and decision-making. This could lead to competitive advantages and disruption in many industries.
• Broad adoption of the recommendations could accelerate AI innovation and automation. This may displace certain jobs but also create new roles to manage data and AI systems.
• Effective governance and automation will be crucial as data volumes and complexity grow. Companies that don’t adapt may struggle with costs, security risks, and compliance.

Alternative perspectives

• The analysis relies on Snowflake customer data, who may not represent the broader range of organizations. Findings could be biased.
• Growth metrics may be tied to Snowflake’s expanding customer base versus behavioral changes in how data is used.
• Focus is on operational metrics versus business impact. More study is needed to quantify benefits.

AI predictions

• Generative AI adoption will surge as models leverage unified, well-governed data platforms rather than fragmented sources.
• Automation of data platform management will expand as AI recommends optimizations based on usage patterns.
• Regulations around data transparency and algorithmic accountability will increase as AI permeates business and society.

Glossary

• Data Cloud: Snowflake’s platform that unites siloed data and enables organizations to discover, securely share, and execute diverse analytic workloads on a single integrated platform.
• Powered by Snowflake: A Snowflake program that helps SaaS providers build, support and scale applications in the Data Cloud rather than re-siloing data.
• Snowgrid: Snowflake’s unique cross-cloud technology layer that helps global organizations collaborate across regions, clouds, and geographies by enabling a single, integrated experience.
• Stable edges: In Snowflake, a “stable edge” refers to an ongoing data share between a provider and consumer, indicating continuous value. Stable edges increased 93% indicating more recurring value from data sharing.
• Object tagging: A Snowflake feature that allows users to add context through metadata tags to secure, govern, and trigger actions on data. Tagging increased 40x showing improving governance.
• Dynamic masking: A Snowflake policy that allows sensitive data to be concealed from unauthorized users down to granular levels, enabling secure sharing. Masking policies grew 205% highlighting governance rigor.
• Automated warehouse resizing: A Snowflake feature that automatically scales compute resources up and down to optimize performance and costs. 71% growth in resizing shows preference for automation.