Omni Group devs explain how they use Apple Foundation Models
With Foundation Models, Apple has given developers the power to use Apple Intelligence large language models (LLMs) from within their own apps using a few lines of code. This is an important step toward the endpoint AI ecosystem the company is painstakingly working to create.
I spoke with developers from The Omni Group to find out more about what these models can do and how they use them in their apps — specifically CEO Ken Case and legendary evangelist for automation on Apple’s products, Sal Soghoian.
Introducing Omni Automation
The obvious place to deploy foundation models was within Omni Automation, the company’s existing tool for automation across its products. Omni Automation, using Apple Foundation Models (AFMs) and JavaScript, lets users locally generate clean, structured, multi-level data to automate tasks and workflows easily and securely — all within constrained token limits. Thanks to Foundation Models, the team has been able to make access to Apple’s own LLMs available to Omni Automation users through JavaScript. The integration provides structured, multi-level data using schemas (JSON).
In a quick and impressive demo, Soghoian showed how Omni Automation, combined with AFMs, helps users build powerful, customized workflows that use Apple Intelligence within more complex automated workflows.
It’s incredibly textual
During the demo, Soghoian stressed that AFMs can be interacted with using both text-based conversation and JSON schemas, which offers a more powerful and precise way of making AFM requests than using Shortcuts.
He showed how you can use JavaScript in Omni Automation to:
Send a natural language prompt to AFM.
Receive either raw text or structured JSON data back.
Or retrieve detailed multi-layered data (e.g., planet type, distance from the sun) using more advanced schema examples.
He also showed how it becomes possible to use the intelligence to plan projects, suggest routines, estimate tasks, plan, and more. The combined solution functions more as an intelligent assistant within user-defined workflows than a free-roaming AI agent.
There are some restrictions. Apple currently limits prompts to 4,096 tokens, but this can be extended by handing off some tasks to other apps and services. The limit is due to hardware constraints and efforts to keep the model lightweight for all users.
(It somewhat reflects computing’s early days: “I can remember when we had 4MB of memory in Macs,” they agreed.)
“You can provide the foundation models with tools that it can call when it doesn’t know how to do something…. Each tool can actually be a perfect oracle, unlike the language model, where it can just make up answers,” said Soghoian.
What happens in AFM, stays in AFM
The thing is, all the automation takes place on the device. Case and Soghoian are advocates for this approach, as it means their customers can work with confidential information while maintaining complete privacy. Given the cost of computers, components, and computing, less is most certainly more, so it makes sense to bring AI services out of the cloud and onto the device, which is precisely what Apple’s approach seems to be.
“The exciting thing about what Apple is doing from our point of view with these Foundation Models is that they’re running on the silicon that is already at our fingertips, right?” said Case. “We’re not going off and hitting somebody else’s server in the cloud and using who knows how much energy to do whatever it’s trying to do.
“We know exactly how much energy our laptops are using because we charge them and we know all of the data is being maintained locally on our device, so it’s secure. There’s no risk of, you know, some confidential information being used to train somebody on this data, and who knows what happens next.”
That matters when you make professional-level products.
OmniFocus, for example, is a task management solution aimed at pro users who want the benefit of AI to help them structure tasks and time, but don’t want that information exfiltrated, or used to fuel platoons of tedious follow-up advertising. “I don’t want to all of a sudden be on everybody’s phone list because I wanted to know how to add solar power to my home,” said Soghoian.
Privacy and AI
Privacy is important to the ultimate expression of AI. Public attitudes to privacy are changing as people who once accepted that companies like Google tracked their searches in exchange for better services become more cautious and protective of their data. Omni Group’s approach mirrors Apple’s own outlook, which is to build software that keeps users in control of their data. That means data can stay entirely on the device, and any information selected to sync with Omni’s servers is end-to-end encrypted so Omni cannot read it. Users can also host their own WebDAV sync servers.
Case, whose company maintains its own tight privacy controls, said he remains optimistic about privacy. He doesn’t believe it will be lost unless everyone stops trying to protect it. Apple’s public stance on the matter has helped maintain the conversation on the matter.
The power of Apple Silicon
Soghoian led Apple’s own automation efforts for years. He observed that with Apple Intelligence, Apple is now able to implement ideas the team originally explored years ago because of the sheer power and performance of Apple Silicon. “Knowing what I know about what we used to do,” Soghoian said, “I think that the next stage is going to be very exciting, and people will forget about everything else once the last pieces come into play.”
For Soghoian, the move for Siri to become capable of “weaving a needle” through all the hoops of app intents will enable Apple Intelligence to get things done for you just by asking for it to be done. “We had that type of approach prototyped, but we used libraries because we did not have a concept of app intents,” he said.
Apple Intelligence’s LLMs used with Omni Automation lets you use those AI tools from within your own tasks. None of which would be possible without Apple Silicon. “It takes a lot of processing power to handle the myriad calculations required,” he said.
“This is already building a really great foundation for creating self your own set of intelligence tools that keep data with you. My data isn’t passed to any third party — it stays on my machine, and I can use it as I need to.”
You can follow me on social media! Join me on BlueSky, LinkedIn, and Mastodon.Omni Group devs explain how they use Apple Foundation Models – ComputerworldRead More