--- title: "Hitting AI Boundaries with Edge Technology" newsletter: "MLOps Community" date: 2025-01-23 source: https://aaif.live/newsletters/mlopscommunity/2025-01-23-hitting-ai-boundaries-with-edge-technology --- # Hitting AI Boundaries with Edge Technology *Plus, is fine-tuning all that fine?, What's up doc? Tips for writing better documenation, hidden gems, Slack spotlight, and the sandbox.* *MLOps Community — Agentic AI Foundation, 2025-01-23* Let's do a thought experiment - I want to hear how you would reason through this: Assuming money were not a constraint, what would the global demand of GPUs be? Said differently, how many GPUs would the world buy if they could have as many as they wanted? ## Efficient Deployment of Models at the Edge Efficient Deployment of Models at the Edge Krishna Sridhar // VP of Engineering @ Qualcomm If you’re stumped by edge AI, you’ll be bowled over by this episode. Alongside sharing a project that uses a smartphone for real-time cricket ball tracking, Krishna offered fascinating insights on edge AI innovations and Qualcomm’s AI Hub, which makes deploying AI models on edge devices simpler. The platform allows developers to train models using tools like PyTorch and deploy them efficiently on Qualcomm chips or other hardware, with automated optimization and performance checks. One standout point was how Qualcomm handles model deployment across chip generations: * Developers can create a single model that adapts to various chip versions, ensuring compatibility while leveraging specific hardware features for better performance. * The system automatically detects chip features and maps computations accordingly, offering seamless scalability across devices. Krishna also touched on real-world applications like real-time sports tracking apps and on-device LLMs for summarization, showcasing how edge AI is pushing boundaries in compute efficiency, memory management, and use case versatility. Have a listen, and you’ll be doing more than pushing the edge - you’ll be hitting boundaries in no time.* *That’s like hitting a homerun! Video [https://go.mlops.community/7rbtr1] || Spotify [https://go.mlops.community/lwvs8b] || Apple [https://go.mlops.community/l4nazw] ## Want to Write Better Docs? Here’s What GenAI Can Do for You Want to Write Better Docs? Here’s What GenAI Can Do for You With thanks to Nehil Jain for their contribution. It’s ironic that just the thought of docs can make you feel ill. To help overcome your documentation nausea, this blog explores how generative AI can help with technical documentation – particularly for MLOps workflows. It looks at practical remedies that make documentation more efficient and user-friendly, including: * AI tools can automate the most painful parts, like generating API references and code examples, providing instant relief from repetitive tasks * Tools can adapt documentation for different audiences, making complex concepts digestible for non-technical users while maintaining the full technical potency for engineers * Integration with platforms like Notion or Confluence can keep workflows healthy and ensure updates are centrally maintained It also highlights AI’s ability to diagnose documentation gaps by analyzing logs and user feedback, leading to clearer pipelines and smoother onboarding. Have a read, and soon you’ll be writing ‘sick’ documentation, as the kids probably don’t say anymore. Read it here [https://go.mlops.community/pica6y] ## Hidden Gems ## Snow (Hey Oh) // Snow (Hey Oh) // Gem [https://go.mlops.community/yagknv] // Song [https://go.mlops.community/exvq5i] A blog post outlining Canva’s adoption of Snowpipe Streaming for real-time data ingestion, highlighting the shift towards a continuous data platform to support faster analytics and improved decision-making across their ecosystem. Clear The Air // Gem [https://go.mlops.community/s87m4n] // Song [https://go.mlops.community/5mcqjq] An open-source GitHub repository for AIWattch's Chrome extension, designed to monitor carbon emissions during ChatGPT interactions, providing users with real-time insights into the environmental impact of their AI usage. Little Differences // Gem [https://go.mlops.community/lvjp3q] // Song [https://go.mlops.community/e5dj9u] A LinkedIn post outlining differences in fine-tuning LLMs for deterministic vs. freeform tasks, highlighting strategies for evaluation, temperature settings, and cost optimization. Keep It Good // Gem [https://go.mlops.community/r7zcxx] // Song [https://go.mlops.community/92ghw9] A survey examining risks in LLMs, including privacy leakage, hallucinations, and malicious use, while proposing mitigation strategies to enhance real-world application performance and ethical deployment. ## Slack Spotlights Slack Spotlights Stephen Oladele [https://go.mlops.community/733wfg] shares some of the chat you might have missed ## Mia Suniaprita This week, Mia Suniaprita asked the community [https://go.mlops.community/m5xwxl] for recommendations on the best tech stack to build a Retrieval-Augmented Generation (RAG) system that works with scanned PDF documents. Mia noted that the stack includes GCP VertexAI [https://go.mlops.community/57iv1n] and Azure Databricks [https://go.mlops.community/o35qu0]. Here's a deeper look at what the community had to say: 💡 Key Recommendations: * Notes on the Tech Stack Notes: Folks within the community suggested that Databricks might be too barebones for RAG compared to MaaS (model-as-a-service) options like Gemini [https://go.mlops.community/2vstx0]. * Late-Interaction Embedding Models: Both Patrick Barker and Naaman Newbold highlighted late-interaction models like ColBERT [https://go.mlops.community/8v1k2x] and ColPali [https://go.mlops.community/74f58w] as state-of-the-art solutions, particularly for advanced semantic search. * Extracting PDF Content: Médéric recommended starting with content extraction, especially for scanned PDFs, using tools like Google Document AI [https://go.mlops.community/mposvw] for OCR support. Médéric also suggested loading content into Gemini 1.5 Flash (with prompt caching for reduced latency and costs) or the new Vertex RAG Engine [https://go.mlops.community/b0bscl], which uses tools like Document AI for OCR tasks to handle scanned PDFs effectively. * PDF Complexity: Alex Strick van Linschoten pointed out that your choice of tools might depend on the complexity of your PDFs—whether they’re plain text, multi-column, or filled with charts and graphics. ColPali may help when handling complex layouts. 🔧 Helpful Tools Shared: * LlamaIndex and VertexAI models [https://go.mlops.community/e5gu39] for building custom pipelines. * Unstructured.io [https://go.mlops.community/isvoj3] for preparing data for LLMs. * AgentBuilder [https://go.mlops.community/ducrx3] for easy setup and testing (e.g., integrations with Google Chat and Web UI). The integration options may be limited here, so check to be sure. * Marly [https://go.mlops.community/vlmhvg] and Docling [https://go.mlops.community/ajdcgj] (shared by Bartosz Mikulski) as additional tools to explore for document preparation. ❓ Key Questions to Consider: * Steven Wu: Are the scanned PDFs searchable (contain actual text) or entirely image-based? * Alex Strick van Linschoten: Does the content include embedded charts or graphics requiring additional processing? The consensus? Focus on OCR for scanned content first, then explore frameworks like Llama Index, Vertex AI Models, or embedding-centric tools based on your needs. Late-interaction models like ColBERT are a strong contender for refining retrieval. Missed the chat? Jump into the conversation on Slack [https://go.mlops.community/m5xwxl] and share your insights! ## Job of the Week [https://go.mlops.community/6ng7qe](https://go.mlops.community/6ng7qe) ## The Sandbox The Sandbox A little place to test some ideas Trying out a few things here - let us know what you think here [https://go.mlops.community/hx6agg] or email steve@mlops.community Back to the Feature A highlight from last week Ankur's blog comparing ZenML, Flyte, and Metaflow resonated as it broke down how each ML pipeline orchestration tool handles workflow management differently. Through practical comparisons of features and real-world use cases, it helped teams navigate choosing their ideal orchestration solution. Read it here [https://go.mlops.community/xhjuix] Gatewaze Grooves Sharing music picks from our latest members through the Gatewaze [https://go.mlops.community/GwazeEmail]. Loving the replies that you guys are sending in about your favorite bands and artists. It’s been great for reminding me of some I’ve not listened to in a while, like Greensky Bluegrass [https://www.youtube.com/watch?v=_mDUislbLiM], and Jon Hopkins [https://www.youtube.com/watch?v=R1MFxzZwrb0], plus hearing new (to me!) stuff, like Underworld [https://www.youtube.com/watch?v=XiMrrleH_hI]. There’s too many to list here, so if you’re after some inspiration on what to listen to next, there’s a list here [https://go.mlops.community/Grooves], and I’ll finish with this video [https://www.youtube.com/watch?v=fPYm5cjP23Q] that joins the old and the new! Tech Teaser A mini MLOps mindbender It's been a long week/month/year, and you know alcohol isn't the answer but it is a solution. You create an AI bartender that generates cocktail recipes. Adding one more ingredient increases the number of possible combinations exponentially. If there are n ingredients, how many combinations are possible? Click here [https://go.mlops.community/TechTeasers] for the answer. Interested in partnering with us? Get in touch: partners@mlops.community Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin]. The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/]. --- Source: https://aaif.live/newsletters/mlopscommunity/2025-01-23-hitting-ai-boundaries-with-edge-technology