On-device AI or AI as the OS Changes Everything

10 min readApr 27, 2024

TL;DR:

The availability of on-device AI is poised to magnify the impact of AI on society significantly. This leap forward promises near-instantaneous data processing, enhanced privacy, and personalized user experiences without continuous cloud connectivity. It’s a shift that will result in more intelligent, more intuitive technology closely integrated into our daily lives, offering a 10x impact compared to the current cloud-based AI models. While it heralds exciting possibilities in efficiency, personalization, and accessibility, it also brings challenges that require careful navigation to ensure ethical and equitable use. Read more to find out which company will lead the On-device AI revolution and why On-device AI will increase educational inequality.

Listen to the Podcast.

Three rings that enable AI as the OS — Small Language Models, Powerful Processors and AI as OS by Greg Twemlow — Three rings that enable On-device AI — Small Language Models, Powerful Processors and AI as OS by Greg Twemlow

I was thinking about the recent developments in AI, specifically the advancements by companies like Apple and Tesla. I realized they’re a marker of the swift progression towards an “AI as the OS” future. With these AI developments, the concept of AI as a layer or an application and as the fundamental operating framework of devices is more tangible. It also caused alarm bells to ring loudly.

It will almost certainly herald increased societal inequality, given that On-device AI will only be available to those who can afford more expensive devices.

Apple’s On-Device AI

Apple’s new OpenELM* is a significant move, indicating a shift towards more private, efficient, and potentially more responsive AI that runs directly on users’ devices. This can manifest in several scenarios:

Enhanced Personal Privacy: As AI tasks are processed on the device, reliance on cloud servers diminishes, enhancing user privacy and data security.
Improved Functionality: On-device AI will offer users continuous and faster AI interaction, regardless of network connectivity. This will enable more sophisticated and seamless integration of AI in daily tasks.
Ecosystem Synergy: The ability to fine-tune and infer efficiently on Apple devices suggests a potential ecosystem where all devices, from iPhones to Macs, are AI-enhanced, offering a unified experience.

*OpenELM source code along with pre-trained model weights and training recipes is available at https://github.com/apple/corenet.

Tesla’s AI Ecosystem:

Tesla’s ecosystem (explained in the image below) shows an integrated approach where various elements, including vehicles, energy solutions, and now humanoid robots, are controlled by AI. This offers a glimpse into a world where AI doesn’t just perform tasks — it manages ecosystems.

Robots as an Extension: Tesla’s Optimus robots, integrated into everyday life, represent an AI-operating system in a mobile, humanoid form. This could change how we perceive and interact with robots, transitioning them from tools to partners.
AI-Driven Value Creation: Musk’s assertion that Optimus contributes to most of Tesla’s long-term value signifies the anticipated impact of AI as the core of business models, potentially extending well beyond traditional vehicular and energy solutions.

Societal Implications

The stratification of AI usage presents societal implications that need careful consideration:

AI-Class Society: Different levels of access to AI — free, paid, and on-device — will likely lead to increased inequality in which the ‘AI-class’ could have significant advantages over the ‘non-AI class’ regarding productivity, learning, and quality of life.
Economic Impact: This stratification could affect employment and economic participation, as those with advanced AI capabilities might outperform others in the job market.
Educational Divide: The disparity in access to AI could also widen the academic gap, with those having on-device or superior AI access gaining an edge in personalized learning and skill acquisition.

Scenarios for Businesses and Individuals

Strategic Adaptation: Companies and individuals alike must strategically adapt to this changing landscape, potentially rethinking how they operate, compete, and collaborate in a world where AI can take on complex, creative, and decision-making roles.
Ethical and Regulatory Frameworks: Robust ethical and regulatory frameworks that guide the development and deployment of AI are imperative to prevent deepening inequalities and ensure that benefits are distributed fairly.

The “AI as the OS” concept is accelerating towards reality. Its impacts will reverberate across all facets of life, demanding foresight, adaptability, and a commitment to steering this transformation towards society’s greater good.

Generative AI based on large language models (LLMs) has revolutionized access to and processing of knowledge. This shift is akin to the leap from printed encyclopedias to the dynamic and interactive world of the Internet. With the onset of on-device AI, we’re standing on the cusp of another quantum leap in AI’s impact. Here’s why the on-device AI could signify a 10x increase in AI’s influence:

Accessibility and Speed:

Instantaneous Interaction: On-device AI can process and respond to queries in real-time without the latency associated with cloud processing.
Continuous Availability: Unlike cloud-based systems, on-device AI does not require an internet connection, making it perpetually accessible.

Privacy and Security:

Data Control: Processing is done locally, so sensitive data doesn’t have to leave the device, giving users greater control over their privacy.
Security: Reducing dependency on the cloud mitigates risks associated with data breaches and unauthorized access.

Personalization:

Tailored Experiences: On-device AI can learn and adapt to individual user preferences and behaviors without compromising privacy.
Contextual Awareness: Devices can utilize sensor data (location, movement, etc.) to offer highly contextual and situationally aware assistance.

Efficiency and Sustainability:

Energy Savings: Local processing can be more energy-efficient than transmitting data to and from the cloud.
Reduced Bandwidth: On-device AI conserves bandwidth and reduces network congestion by minimizing the need for data transmission.

Functional Intelligence:

Advanced Capabilities: Devices will perform tasks, anticipate needs, and proactively offer solutions.
Cognitive Enhancement: On-device AI has the potential to act as an extension of the human brain, augmenting memory, calculation, and reasoning skills.

Innovation and Market Dynamics:

New Business Models: On-device AI can enable innovative services and features, creating new markets and disrupting existing ones.
Competitive Edge: Companies implementing on-device AI can gain significant user experience and performance advantages.

Societal Impact:

Educational Transformation: Democratizing personalized learning through on-device AI will reshape education.
Healthcare Revolution: On-device AI could lead to personalized healthcare monitoring and diagnosis, becoming a constant health companion.

Given these transformative potentials, it’s reasonable to anticipate that integrating AI into our daily devices will amplify its influence exponentially. This obviates the complexities and challenges that such a profound change entails, such as ethical considerations, the digital divide, and the need for robust frameworks to manage this transition responsibly.

The future shaped by on-device AI holds immense promise. It requires thoughtful stewardship to ensure its positive and equitable impact.

Apple as the On-device AI Leader

Apple’s unique position as a company with control over its hardware (Device Stack) and software ecosystem (App Stack) gives it a distinct edge in being the first major company to deliver on-device AI or “AI as the OS.” Here are several factors that support this assessment:

Hardware and Software Integration: Apple’s vertical integration allows for the tight coupling of hardware and software, which is crucial for optimizing the performance of on-device AI systems.
Processor Development: Apple’s track record with its custom silicon, like the M1 and A-series chips, demonstrates its capability to produce powerful, energy-efficient processors that could support the demands of on-device AI.
Privacy Focus: Apple has consistently emphasized user privacy. This stance aligns well with AI’s on-device processing, keeping sensitive data local to the user’s device.
Ecosystem Control: Controlling the App and Device Stacks ensures that Apple can push updates and new features across its entire device lineup without depending on third-party hardware constraints.
Market Positioning: Including an advanced Small Language Model in the next iPhone could reinforce Apple’s market position by offering cutting-edge technology that enhances user experience and sets a new industry standard.
Innovation Reputation: Apple has a history of pioneering new technologies in consumer devices. Introducing on-device AI as a core feature would align with their innovation and user-centric design strategy.

Considering these factors, Apple is well-positioned to be a frontrunner in transitioning towards on-device AI. It could announce these advancements at its 2024 Worldwide Developers Conference (WWDC).

However, this space is highly competitive, and other tech giants with substantial AI research and development may also be approaching similar milestones. It will be interesting to see how the race unfolds and which company ultimately crosses this pivotal threshold first.

Non-Obvious Impacts of On-device AI

Exploring the non-obvious impacts of on-device AI, particularly in education, where the impact could be revolutionary.

In reality, On-device access to AI won’t be universal, which will likely exacerbate the inequality of educational opportunity, meaning lower-income students miss out on enhanced learning by using On-device AI.

If K-12 students can access on-device AI, it could significantly enhance their ability to comprehend and apply mathematical concepts. Here’s how:

Personalized Learning: On-device AI could tailor instruction to each student’s learning pace and style, providing personalized explanations and practice problems, thus improving comprehension.
Interactive Problem Solving: AI could guide students through problem-solving steps interactively, offering hints and explanations and adjusting difficulty in real time based on the student’s performance.
Instant Feedback: Students could receive immediate, constructive feedback on their work, allowing for a more dynamic learning process and the ability to correct misunderstandings quickly.
Accessibility: On-device AI can make advanced tutoring accessible to a broader range of students, including those in under-resourced schools who may not have regular access to human tutors or advanced math courses.
Visualization and Simulation: AI could generate visualizations for complex mathematical concepts, making abstract ideas more concrete and understandable through interactive simulations.
Language Processing: On-device AI can interpret natural language, enabling students to ask questions in their own words and receive explanations that make sense to them, thereby breaking down barriers to understanding.
Enhanced Collaboration: AI could facilitate group work by connecting students working on similar problems or concepts, even suggesting collaboration when it detects complementary skills or knowledge gaps.
Augmented Reality Learning: Integrating AI with AR could bring math into the physical world, helping students understand geometry, for example, by visualizing shapes and volumes in their environment.
Gamification: AI can turn learning into a game, providing a competitive element that can motivate students and reinforce concepts through play.
Data-Driven Insights: On-device AI can track progress over time, offering insights into learning patterns, predicting potential challenges, and recommending focus areas for improvement.

The impact of on-device AI in education could extend beyond mathematics, potentially transforming all areas of learning by making education more personalized, engaging, and effective. This level of individualized learning support was previously unimaginable at scale, but with AI advancements, it is possible. It could also lead to low-income families being left far behind those who can afford On-device AI.

A Generation of Mathematically Competent Students

If nearly every K-12 student were to achieve mathematical competence through on-device AI, it could lead to many transformative changes in society.

Economic Growth and Innovation

Skilled Workforce: A generation of mathematically competent individuals would contribute to a workforce adept in problem-solving and critical thinking, driving innovation and productivity in various industries.
STEM Fields Expansion: Fields like engineering, technology, and science might see an increase in qualified professionals, fostering advancements in research and development.

Educational Advancement

Higher Educational Achievement: With stronger math foundations, students might pursue more advanced education and research, pushing the boundaries of knowledge.
Curriculum Evolution: Educational curricula could shift to more complex, creative, or interdisciplinary studies earlier, as basic mathematical competencies are mastered more quickly.

Societal Benefits

Informed Decision-Making: A society with a strong grasp of mathematics might make more informed decisions on personal finance, health, and civic engagement.
Reduced Inequality: On-device AI could level the educational playing field, offering high-quality resources to students regardless of socioeconomic background.

Technological and Data Proficiency

Data Literacy: As mathematics underpins data analysis, a mathematically competent society would likely be more adept at understanding and using data, which is increasingly important in a data-driven world.
Tech-Savvy Citizens: With the technological interaction required for on-device AI, individuals might become more comfortable with emerging technologies, contributing to a tech-savvy populace.

Global Competitiveness

International Leadership: Countries with widespread mathematical competence in their youth could lead in global competitiveness and economic leadership.
Innovation Hubs: These societies might become hotbeds for startups and innovation, as a mathematically skilled population could drive entrepreneurial endeavors.

Impact on Other Academic Disciplines

Interdisciplinary Approach: With a strong math foundation, students may engage more deeply with other academic disciplines, from physics to economics, leading to a more holistic and interconnected approach to education.

Challenges and Adjustments

Workforce Shifts: As the need for routine mathematical computation decreases, there could be a shift towards jobs requiring more advanced, creative, or human-centric skills.
Continuous Learning: The pace of change may require a societal shift towards lifelong learning to keep up with advancements and applications of mathematics in various fields.

The widespread enhanced mathematical competence facilitated by on-device AI could catalyze progress across many dimensions of society, driving forward individual capabilities and collective societal development.

About the author: Greg Twemlow, Co-Founder of Future Skills Studio and Member of Medium 200-Club.

Greg Twemlow : fostering interdisciplinary dialogues and continuously exploring the interplay between technology, particularly GenAI, education, ethics, and philosophy, and, since 2018, CEO of Future Skills Studio, focused on experiential learning programs for young people.