How this online gifting site protects customer privacy with Meta’s AI tools

Innovative privacy protection in AI-driven gift recommendations: Untukmu.AI, an Indonesian personalized gift recommendation platform, has developed a unique approach to safeguarding customer data privacy using Llama on edge devices.

• The company has created a semi-decentralized personal assistant that prevents direct access to customer data while still providing personalized recommendations.
• This innovative solution addresses growing concerns about data privacy and the protection of Personally Identifiable Information (PII) in the post-cookie era.

Technical implementation of split inference: Untukmu.AI’s Senior ML Engineer, Puja Romulus, devised a split inference approach to balance privacy concerns with computational feasibility.

• Customer data is stored only on edge devices and user accounts, ensuring the company doesn’t have direct access to sensitive information.
• The Llama 3.1 8B model is split across two checkpoints: the first transformer layer runs on edge devices, while the remaining 31 layers and output layer operate in the cloud.
• This approach allows the use of the original model without quantization, maintaining output quality while addressing privacy issues.

Advantages of Llama for privacy-preserving applications: The Untukmu.AI team selected Llama 3.1 8B after evaluating several open weight models for their specific needs.

• Llama 3.1 8B offered an optimal balance between output quality and resource efficiency for entity extraction and product recommendation tasks.
• The model performed well without requiring fine-tuning, making it an attractive choice for the company.
• The potential for future multimodal versions of Llama, such as the rumored 405B model, further solidified the decision to use this technology.

The split inference process: The innovative approach developed by Untukmu.AI involves a multi-step process to protect user data while generating personalized recommendations.

• Predefined prompts from Untukmu.AI or third parties are sent to users and merged with their personal data on edge devices.
• The merged prompt is processed at the edge to generate a first-layer tensor, which is then sent to the server for further computation.
• This iterative process continues until a stop token is detected, at which point the server converts the token list into full text.
• The final output is sent back to both the user and the third party, ensuring transparency and data privacy throughout the process.

Data visibility and user control: Untukmu.AI’s approach prioritizes user control and transparency in data handling.

• Users have full access to all their information and can monitor how their data is used.
• The company only has access to non-personal data, while third-party providers can view their own prompts and the resulting output but not customers’ personal data.
• This policy ensures that sensitive information remains protected while still allowing for personalized recommendations and services.

Potential applications and future directions: The split inference approach developed by Untukmu.AI has broad implications for industries dealing with sensitive user data.

• Sectors managing large volumes of unorganized data while needing to protect sensitive information can benefit from deploying Llama for split inference processing.
• Untukmu.AI is focused on implementing split inference with larger Llama models to further improve output quality while maintaining strong privacy protections.
• The company continues to explore innovative ways to safeguard customer data, recognizing the growing importance of privacy-preserving technologies in the AI-driven marketplace.

Broader implications for AI and privacy: Untukmu.AI’s innovative approach to privacy protection in AI-driven recommendations highlights the evolving landscape of data privacy and AI implementation.

• As concerns about data privacy continue to grow, solutions like split inference may become increasingly essential for companies looking to leverage AI capabilities while respecting user privacy.
• This approach could potentially set a new standard for privacy-preserving AI applications, encouraging other companies to develop similar solutions that balance advanced AI capabilities with robust data protection measures.
• The success of Untukmu.AI’s implementation may also spur further research and development in privacy-preserving AI technologies, potentially leading to new breakthroughs in this critical area of AI ethics and implementation.