Engineering AI for Creativity and Curiosity on Mobile

Google’s strategic approach to integrating advanced artificial intelligence into consumer mobile experiences was recently highlighted in a detailed presentation by Bhavuk Jain, a key engineer behind two prominent features: AI Wallpapers and Circle to Search. Jain’s insights underscored the meticulous process of transforming foundational AI models into robust, user-facing products, emphasizing architectural rigor, user-centric design, and an unwavering commitment to safety and scalability. His presentation, delivered at a recent industry event, offered a rare glimpse into the complex engineering challenges and innovative solutions that underpin Google’s latest mobile AI offerings.

The Foundational Pillars of Modern AI Productization

Modern AI product development, as articulated by Jain, rarely begins from scratch. Instead, it leverages massive foundational models, which, despite their power, are initially "unpredictable and unconstrained." The journey from a raw model to a viable product is a multi-stage architectural endeavor comprising four critical, non-negotiable steps: post-training, fine-tuning, retrieval and grounding, and finally, inference, guardrails, and deployment.

Post-training: Aligning AI with Human Preferences
The initial phase, post-training, is likened to teaching a brilliant but raw student manners and safety. Its primary goal is to align the powerful base model with human preferences, aesthetic styles, and essential safety protocols. This stage predominantly utilizes two families of techniques: Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO). While RLHF offers flexibility, it involves managing a complex reward model and an intricate reinforcement learning loop. DPO, conversely, is simpler and more scalable, operating directly on preferred versus rejected outputs through a supervised learning objective, thereby skipping the separate reward model. Regardless of the chosen technique, rigorous evaluation is paramount, requiring clear, measurable criteria for human raters focusing on aspects like format, quality, completeness, and accuracy. This early architectural decision balances flexibility with scalability, locking in the alignment before proceeding.

Fine-tuning: Specializing the Generalist Model
Once aligned, the foundational model remains a generalist. Fine-tuning transforms it into an expert tailored for specific use cases. This can involve standard supervised fine-tuning, where the model learns specific behaviors from large datasets of input-output pairs (e.g., question-answers). More recently, Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA, QLoRA, and adapters have emerged as a "modern sweet spot." These methods allow for the training of tiny, specialized modules that augment the large, mostly frozen base model, drastically reducing training costs and time. Jain also noted a crucial point: sometimes, fine-tuning isn’t the answer. If the model lacks fresh, factual, or private knowledge, the solution lies in grounding, not more training. The strategic choice here involves balancing specialization and new behavior against the increased complexity, maintenance, and financial costs of extensive fine-tuning.

Retrieval and Grounding: Battling Hallucinations
A critical and increasingly important step is retrieval and grounding, a layer designed to keep the AI system accurate, factual, and up-to-date. Instead of relying solely on the model’s parametric memory, which is fixed at training time, this phase connects the AI to external "sources of truth." These sources can range from structured knowledge bases and fresh web results to on-device context. This significantly reduces the phenomenon of "hallucinations," where AI generates plausible but incorrect information, enabling the model to provide answers rooted in real knowledge rather than guesswork. This is where modern AI systems integrate search technologies, embeddings, Retrieval-Augmented Generation (RAG) frameworks, and external tool utilization.

Inference, Guardrails, and Deployment: Ensuring Safety and Scale
The final stage involves wrapping the aligned, specialized, and grounded model in a robust layer of deployment, infrastructure, and, most importantly, a multi-layered safety architecture known as guardrails. These mechanisms constrain what the model can see, say, and do, acknowledging that even fine-tuned models can exhibit unpredictable behavior.

Guardrails are implemented across multiple fronts:

• Data Guardrails: Filtering sensitive or toxic content from training and evaluation data.
• Model Guardrails: Using safety-tuned base models, strong system prompts, and separate safety classifiers to reshape raw model output.
• Runtime Guardrails: Real-time inspection of inputs and outputs during inference to detect personally identifiable information (PII), toxicity, or "jailbreak" attempts, alongside strict control over external tool access.
• Governance and UX: Human safety nets including logging, audits, human review for high-risk actions, and user-facing controls like reporting and feedback.

Deployment, the "moment of truth," focuses on running the model reliably at scale. Key considerations include unified multimodal serving, handling diverse inputs like text, images, audio, and video through a single API; optimizing for latency, cost, and scaling through techniques like batching, caching, and autoscaling to manage expensive GPU/TPU resources; and robust observability and control via rich logging and metrics. This infrastructure stack is the bridge between a trained model and its daily use by millions of users without incident.

Case Study 1: AI Wallpapers – The Generative Challenge of Creativity

AI Wallpapers, a feature enabling users to create unique, high-quality phone wallpapers instantly from a description, exemplifies the generative phase of AI engineering, optimized primarily for creativity. Launched approximately two years prior to the presentation, its genesis was rooted in a clear user need and Google’s design philosophy.

The "Why" Behind AI Wallpapers:

1. Personalization and Material You: The feature offered an immediate user benefit—deep personalization. It seamlessly aligned with Android’s Material You design system, where the phone’s interface adapts its color palette to match the wallpaper, creating a uniquely personal experience.
2. Massive User Need: Data indicated that changing wallpapers is the most popular way users customize their phones, often spending significant time searching for the "perfect" image.
3. Technological Readiness: Generative AI provided the capability to deliver an "infinite range of high-quality content," liberating users from static galleries and empowering them to become artists.

Core Challenges in Development:
The AI Wallpapers team tackled four major challenges:

1. UX Challenge (Prompt Engineering): Prompt engineering is complex. Asking typical users to craft sophisticated prompts like "cinematic lighting, 8K, ethereal watercolor" was deemed a recipe for failure. The challenge was simplifying this complexity into an intuitive, fun, and fast mobile UI.
2. Quality Challenge: Generative models lack inherent "good taste," potentially producing technically correct but artistically messy or "weird" images. The goal was consistent artistic control.
3. Safety Challenge: For a feature on millions of consumer devices, preventing inappropriate, harmful, or copyrighted content generation was a "non-negotiable day one problem."
4. Viability Challenge (Cost): Early server-based models were powerful but expensive. As user adoption grew, the operational cost became a significant concern, necessitating cost-efficiency measures.

The "Non-Obvious Bet" on Guided Creation:
The team made a counterintuitive decision: instead of offering an open "blank text box" for maximum freedom (which often leads to user frustration and difficulties in setting reliable guardrails), they chose a "restricted approach" with guided prompt building. This "hybrid model" guided users in crafting effective prompts while still allowing for unique ideas. This decision, validated by user experience research, transformed the team’s role into a "creative partner," prioritizing simplicity and consistent quality over unbridled complexity. This approach also made the engineering challenge of risk assessment tractable, ensuring safe outputs.

Engineering Delight and Robustness:
Within the constraints of a guided experience, the team heavily invested in "delight." This included subtle UI animations (e.g., wavy text, horizontal scrolls, fade-up effects) that made the interface feel alive. Delight also extended to handling failures, replacing harsh error messages with playful visuals, such as a T-Rex attempting to paint, turning potential frustration into amusement. This focus on delight, combined with the restricted input, directly reduced the likelihood and cost of "catastrophic safety failures," allowing the feature to be opened to users aged 13 and above.

High-Level Architecture of AI Wallpapers:
The architecture involves a frontend, a prompt engineering service that translates user selections into highly structured, detailed prompts (often injecting artistic modifiers and negative keywords), and a backend service with the first layer of fast server-side guardrails (classifiers, block lists). Cleared prompts are sent to an inference platform, where a fine-tuned media generation service creates the image. This platform includes a second safety layer—visual classifiers—that analyze the output image for safety before an upscaling step enhances resolution. The finished wallpapers are then sent back to the frontend. This layered approach ensured both quality and safety from launch.

Tackling Quality: The Artist’s Strengths:
The key insight for quality was recognizing that "models aren’t all made equal"; they excel at certain styles. The team identified "sweet spots"—idea clusters (art, animals, people, clothing) where the model consistently produced outstanding results. This involved large-scale testing, generating thousands of images from prompt variations, always concluding with manual human review by designers and researchers. Explicit evaluation criteria (format, quality, completeness, appeal) were developed to make subjective aesthetic judgment measurable. This human-in-the-loop process refined the model’s output to deliver "delightful wallpapers." Learning from these tests, the team designed "suggestion chips" and a structured prompt format, generating and reviewing over a million images to select the most effective keywords for the UI. This structured approach also unlocked inference optimizations like prefix caching, improving efficiency. Collaboration with internal research teams established a powerful feedback loop, enhancing core models and allowing for specific fine-tuning.

Scaling and Global Impact:
Built for a global user base from day one, AI Wallpapers’ success stemmed from relentless UX exploration, rigorous human-rated evaluation pipelines, and a scalable architecture. Instead of a one-off service, it utilized a scalable endpoint reused across multiple Google products, reducing costs and improving reliability. The feature’s launch was met with "incredible" user praise, called "freaking insane" and a "childhood dream," validating the guided creation approach. It achieved true Google scale, integrated across various surfaces and form factors, and adopted by major OEMs like Samsung.

Case Study 2: Circle to Search – The Understanding Challenge of Curiosity

Shifting from generation to understanding, Circle to Search represents another phase of AI engineering. This feature allows users to instantly search for anything visible on their mobile screen without leaving their current application. It is described as a complex, real-time, multimodal retrieval task, translating a visual gesture on a screen into a high-fidelity query.

Motivation: Erasing Friction in Visual Search:
The traditional workflow for visual search involved multiple friction points: taking a screenshot, switching apps, uploading the image, and then searching. Each step was a potential point of abandonment. The "North Star" for Circle to Search was "zero context switching, zero interruption," creating a seamless flow from curiosity to answer. Building on Google Lens’s "world-class visual understanding," the team aimed to deeply integrate AI and achieve deeper system-level OS integration.

Evolution from Google Lens:
Google Lens historically enabled searching the world via camera but required users to stop, open a separate app, and point. Circle to Search is the "next natural step," turning "screen-facing" instead of "world-facing." By replacing multiple steps with a single gesture, it dramatically reduces friction, allowing instant action on curiosity. This shift transformed the task into a real-time retrieval problem demanding extremely low latency and highly relevant results.

Sophisticated Multimodal Architecture:
The architecture of Circle to Search begins on-device, where a user gesture (circle, tap) triggers the capture of a "multimodal prompt"—pixels alongside the structural layout of the device. This rich prompt is fed into an AI pipeline, the core of the AI reasoning. The pipeline then sends the final answer back to the device as a seamlessly integrated results overlay.

AI Pipeline Deep Dive:

1. Core Multimodal Model: The multimodal input hits a fine-tuned model designed to understand the relationship between an image, the user’s question, and the screen’s context.
2. Reasoning and Retrieval: Understanding user intent, the model initiates parallel queries across multiple sources: the Knowledge Graph for structured facts, and web/image search for visually similar items, products, and articles. For example, circling sneakers triggers queries for brand, model, shopping links, and reviews.
3. Synthesis and Delivery: Potential answers are fed into an AI overview synthesis model (the generative part), crafting natural language answers. Before delivery, every answer passes through "safety and responsibility filters"—runtime guardrails ensuring helpful and harmless output.

MUM and Gemini Integration:
The instant, seamless experience is powered by two distinct layers:

• Multitask Unified Model (MUM): Moves the system beyond keyword matching to intent understanding across modalities. If a user circles shoes and types "blue," MUM understands both visual attributes and semantic refinements for precise answers.
• Gemini Integration: Powers richer AI overviews and contextual help, handling complex reasoning and elevating the tool from simple identification to sophisticated problem-solving.

Deep System-Level Integration and Contextual Understanding:
A game-changer for Circle to Search was its deep system-level integration directly into the Android OS. This provides a simple, universal gesture that works everywhere, eliminating the old screenshot-and-upload friction. Crucially, this integration means that when an object is circled, the AI receives a "deeper level of context"—not just a flat image, but the semantic layout of the screen. This allows the AI to pinpoint "exactly which specific pixels belong to the handbag in the entire screen," leading to highly accurate search results.

Ensuring Trust and Accuracy:
Guardrails for Circle to Search focus on trust and accuracy:

1. Factual Grounding: Visual AI is cross-referenced with the Knowledge Graph, a structured, reliable knowledge base, to prevent errors from purely pixel-based interpretations. This is the first line of defense against misinformation.
2. Dedicated Safety Layer: Every synthesized result, especially from generative models, passes through a "final, non-negotiable responsible AI filter" to catch harmful or inappropriate content.
3. Designing for Ambiguity: In low-confidence scenarios (blurry images, obscure objects), the system avoids guessing. A bad answer erodes trust more than no answer. The UX offers broad suggestions gracefully, preventing incorrect information.

Market Impact and Engagement:
The core hypothesis—that removing friction through deep system-level integration would unlock new engagement—proved correct. Circle to Search achieved massive scale, launching on over 300 million devices. This easier access directly contributed to a 70% year-over-year increase in visual searches and became a fast-growing query type, particularly popular with younger users for shopping and translation tasks.

Key Learnings and Actionable Takeaways

The development of AI Wallpapers and Circle to Search yielded distinct but complementary lessons for AI engineering. From the generative AI Wallpapers, the power of clear principles and tight product definitions was paramount in managing the "blank canvas problem." The "non-obvious bet" on guided creation and obsessive evaluation pipelines for artistic quality were critical, as was over-investing in "moments of delight" to make a restrictive experience magical.

For Circle to Search, an AI focused on understanding, the key learning was the transformative impact of deep OS integration. While model improvements were important, the frictionless, seamless invocation made the feature truly compelling for users. This integration also provided a deeper level of intelligence through access to the screen’s semantic layout. Building trust required grounding the visual AI in factual data, cross-referencing model perceptions with a structured knowledge base to ensure reliability.

Bhavuk Jain concluded with actionable takeaways for various roles within the tech industry:

• For AI Engineers: "Productize your AI, not just deploy it." A powerful model is merely the starting point; the real work lies in building the entire system—evaluation pipelines, cost/latency optimizations, and safety guardrails—to create a reliable and trustworthy experience.
• For Tech Leaders: "Champion principle-led development." Difficult decisions, like the guided creation for AI Wallpapers, must be guided by core principles of quality and safety. Leaders must define these principles and empower teams to adhere to them, even when challenging.
• For Product Managers and Designers: "Obsess over the user’s workflow." As demonstrated by Circle to Search, identifying and eliminating friction in the user’s journey is crucial, often as important as the underlying AI itself.

Challenges and Future Outlook

Addressing the challenge of cost, particularly for AI Wallpapers in its early stages (developed about two-and-a-half years ago when models were less efficient), the team implemented several strategies. These included migrating to more efficient models, batching prompts, utilizing prefix caching, and leveraging shared internal endpoints across multiple Google products to reduce both operational and development costs. The project’s development timeline for AI Wallpapers, taking approximately a year, reflects the significant investment in ensuring high quality and safety in nascent AI model environments. The continuous evolution of testing pipelines, driven by human feedback and iterative model improvements, remains central to Google’s AI product strategy.

Google’s endeavors with AI Wallpapers and Circle to Search exemplify a sophisticated approach to bringing cutting-edge AI to everyday mobile users. By meticulously addressing the challenges of model alignment, specialization, factual grounding, safety, and scalability, Google is not merely deploying AI but thoughtfully integrating it to enhance creativity, streamline information access, and ultimately enrich the mobile experience for hundreds of millions globally. These innovations underscore a future where AI is not just powerful but also intuitive, reliable, and deeply embedded in the fabric of mobile interaction.