Every new feature Apple unveiled for iPhone, iPad, Mac, and more this fall (including AI)
Apple's 2024 Worldwide Developer Conference (WWDC) earlier this week shaped up to be one of the company's biggest events in recent years. The opening keynote, which took place on Monday, focused almost entirely on the buzzword we can't stop talking about -- artificial intelligence (AI).
Also: Everything to know about Apple's AI features coming to iPhones, Macs, and iPads
After trailing behind major players like OpenAI, Google, and Microsoft, Apple unveiled a slew of AI features spread across the company's most popular operating systems. While AI was the event's main focus, Apple executives also announced this year's software upgrades for the iPhone, iPad, Apple Watch, Mac, and Vision Pro.
If you couldn't tune into the two-hour-long event, ZDNET has you covered. Here's a complete breakdown of all the announcements from earlier this week.
Apple Intelligence (AI)
Apple unveiled what it calls its new "personal intelligence" system under the name Apple Intelligence. The release puts generative models at the heart of the ecosystem of Apple devices.
With Apple Intelligence, your iPhone can prioritize notifications to ensure you get notified only when it's crucial throughout your day.
The release includes writing tools that leverage AI, including rewriting, proofreading, and summarizing text features available across mail, keynotes, third-party apps, and more.
Users can now create personalized images in the photo library, including sketches, illustrations, and animations. This feature is available in Messages, Apps, Freeform, Keynote, and Pages.
Apple Intelligence can tap into tools and carry out tasks on your behalf, such as "Show me all the photos," "Play the podcast," or "Pull the files that my coworker shared with me last week."
Because it's grounded in your personal information and context, and can retrieve data from across your apps and reference the content on your screen, Apple Intelligence is positioned to be your personal assistant.
Apple emphasized the safety and privacy precautions built into Apple Intelligence, particularly for on-device intelligence processing. The company touted the security of Apple's silicon, A17 Pro, and its M family of chips (M1, M2, M3, and M4).
For tasks that are too large for on-device processing and need to be completed in the cloud, Apple unveiled Private Cloud Compute, which protects users' privacy by running on servers specially created using Apple Silicon. When users make requests, Apple Intelligence first tests on-device capability, but calls on Private Cloud Compute if the task requires more power. Apple reiterated that user data is never stored or sold to external parties.
Siri finally got the AI makeover it deserves, first with a new look: when tapped, light wraps around the edges of your screen. Siri can now better understand users, even if they stutter, due to more advanced natural language processing (NLP). It now has conversational context, remembering what you just said and using it to complete the next task. Users can also type requests to Siri. Because it has in-depth product knowledge, Siri can answer questions about functionality on iPad, iPhone, and Mac. Siri will also have Apple Intelligence's on-screen awareness, allowing it to act on what it sees. The voice assistant can also take actions across apps, including photo editing. With access to your personal context, Siri can understand and complete new commands, such as pulling your driver's license information from a photo and inputting it into a form. The Siri updates are coming to iPad and Mac, too.
Apple Intelligence also powers new features in Mail, including Rewrite, which offers users different versions of what they have already drafted. Suggestions are shown in-line, and Proofread edits for grammar, word choice, and sentence structure. You can also use Summarize to convert your text into bullet points. Smart Reply identifies relevant selections of an email and uses them to help craft a custom message. Summaries will now appear at the top of emails, making browsing an inbox easier. Apple Intelligence can even help prioritize your emails, placing what is most important at the top of your inbox.
There is an all-new focus option: reduce interruptions. When in this setting, your phone will only show you what is most important based on your personal activity and context.
Genmoji allows users to create AI-generated emojis based on what they type. You can also create a Genmoji based on a photo of a friend. Genmojis can be included in-line in Messages and even used for Tapbacks.
Image Playground allows users to leverage AI on-device to create images from text prompts, which can be easily shared in iMessage and elsewhere. The feature is also available in Keynote, Pages, and Freeform, and as a stand-alone Image Playground app.
Image Wand in the Notes app transforms a rough sketch into a polished image and is available directly in the tool palette. For example, you can circle a rough sketch in Notes and open Image Playground to transform your doodle into a fully-fledged image.
Apple Intelligence will also upgrade the Photos app with a new clean-up tool that removes unwanted objects. Search in videos allows users to easily find specific snippets of content, and users can create Memories on-demand, using text to edit and organize photos into movies.
In the Notes app, users can record and transcribe audio, which Apple Intelligence will generate a text summary of. This experience is also available in the Notes phone app.
Apple Intelligence is free on iOS 18, iPadOS 18, and MacOS Sequoia, and will be available to try in English only this summer.
Partnership with OpenAI
Apple also confirmed its partnership with OpenAI by integrating ChatGPT with Siri. With a user's permission, Siri can send a request to ChatGPT for help. For example, if you ask Siri for assistance on a task it deems better for ChatGPT, Siri will suggest you use the chatbot instead and forward your request. ChatGPT's writing capabilities can also be leveraged within certain writing tasks.
Users can access ChatGPT via this integration for free, and OpenAI will not log their data. ChatGPT Plus users can connect their subscriptions to access more advanced features, in which case OpenAI's data usage policies apply.
The ChatGPT integration will be coming to iOS 18, iPadOS 18, and MacOS Sequoia later this year.
iOS 18
iPhone and iPad users will be able to customize their home screen further by placing apps wherever they'd like on the screen, as opposed to the usual fixed grid. App icon colors will also be customizable, allowing users to make apps any color they want or even match their home screen. Users can also change app icons to dark mode.
After five years of remaining untouched, the Control Center received several upgrades, including the ability to customize its toggles, such as flashlight, screen recording, calculator, auto-rotate, screen mirroring, and more, by tapping, holding, and rearranging. The Control Center toggle will also feature different pages with completely customizable user controls, and users can switch controls on the bottom of the Lock Screen.
Apple also added privacy options, including the ability to lock an app, which requires users to authenticate with Face ID or passcode before accessing the app. Users can also hide an app, which makes it disappear from the home screen to a hidden part of their app library.
Messages received several upgrades. Tapbacks, the feature allowing users to react to messages by holding them down, was upgraded to feature different colors and include emojis. Users can add text effects to specific phrases or words instead of the entire phrase. Texts can also be customized further with formatting options like bold, underline, italics, and strikethrough. Lastly, users will be able to schedule messages.
iPhone 14 and later models will have a new Messages via Satellite feature, which allows users to send messages via satellite when they don't have Wi-Fi or cellular service.
The Mail app will automatically categorize emails, a feature that will be available later this year.
The Wallet app now allows users to tap phones together to exchange Apple Cash without requiring them to share personal information like phone numbers.
The Journal app will now show more statistics and insights, including how many entries you've had this year, how many days you journaled, and more.
There is a new Game Mode for iPhone, meant to help gamers optimize their gaming experience. This includes minimizing background activity and using more responsive accessories, such as controllers.
The Photos app got what Apple dubbed its "biggest ever redesign," featuring a cleaner design, a new carousel with highlights that update each day, the ability to pin collections, and an improved search.
The Messages app now supports Rich Communication Services (RCS).
The Safari app was upgraded to include key information about a webpage. You can read more about the Safari upgrades under the MacOS section of this article.
The Calendar app can now pull from the Reminders app for a more seamless schedule overview.
WatchOS 11
The new training load allows users to gain insights into how their workouts' intensity impacts their long-term performance.
The new Vitals app will give users a quick look at their most important health metrics, including heart rate, respiratory rate, wrist temperature, sleep duration, and blood oxygen. It will also provide context to help them make more informed decisions. If something seems out of the ordinary, users will receive pings alerting them of the anomaly.
When a user logs a pregnancy in the Health App on iPhone or iPad, the Cycle Tracking app on Apple Watch is upgraded to show gestational age, and allows users to log symptoms experienced during pregnancy. Pregnant users can also ask to be reminded to take a mental health assessment every month. Using the Walking Steadiness feature, users can also be alerted of increased fall risk.
Users will also experience more customizable Activity Rings, which allow them to pause their rings when they want to take a day off without impacting their award streaks.
Apple Fitness+ was upgraded to include personalized For You tabs, Explore and Library tabs, search features, and enhanced awards.
Smart Stack is also getting more intelligent: it's now able to suggest widgets when needed automatically, among other improvements.
With watchOS, users will get suggestions on the best photo options for their watch face from their photo library based on criteria such as aesthetics, composition, facial expressions, and more. The face can also be customized with different sizes, layouts, and fonts.
Check In is available on Apple Watch, even during workouts, to help users stay safe. Translate is also coming to Apple Watch so users can translate text right from their wrists.
Another notable change is that the double tap gesture can be used to scroll through apps.
iPadOS 18
The update will feature a redesigned tab bar that floats above app content. Users can customize it to showcase their favorite apps and access the most important sections of an app. You can also long-tap the bar to move it around. The tab bar also morphs into the sidebar for added insights.
Shareplay will allow users to remotely control someone else's iPad or iPhone and share drawings on their screens.
In a long-awaited release, iPads will now have a calculator app for the first time, complete with the same interface as the one currently found on iPhones. Plus, users can use it with the Apple Pencil through a new Math Notes experience, which allows users to write expressions that the calculator app will solve for them once they type the equal sign, in their own handwriting,
Handwriting in Notes also got an upgrade with Smart Script, which refines users' writing to make it more legible while keeping the authenticity of the user's handwriting style. The feature can also match copied and pasted text to the user's handwriting. Typed text was also enhanced, with five new highlight colors and the ability to toggle sections under headings or subheadings.
iPadOS 18 supports screen-sharing via SharePlay and the same Control Center customizations, Photos app upgrades, Safari updates, and emoji Tapbacks found in iOS 18.
MacOS 15 (Sequoia)
Apple unveiled MacOS Sequoia, which will include many of the new features that were added to iOS 18 and iPadOS 18, including the updated Safari, Photos, Messages, and the new Passwords management app.
The new iPhone mirroring capability on Mac allows users to experience their phone almost entirely from their Mac. For example, iPhone notifications will now be available on Mac, allowing users to interact with them and open corresponding apps, though the iPhone itself will appear locked.
Window tiling was made possible to help users stay more organized. When users drag a window to the edge of the screen, macOS automatically suggests a tile position.
Video meetings are also getting an upgrade, with new backgrounds and a preview experience that allows you to see what you are about to share before sharing it. This feature works with popular video conferencing applications such as FaceTime and Zoom.
The AI summarization tool will live in Safari to help users process content like web pages and articles more efficiently. Safari will also assist users in discovering more helpful information about a page they are browsing when relevant, such as directions.
Apple also launched a new Viewer experience, which does for video what Reader does for text.
VisionOS 2
Apple unveiled the first major upgrade to its recently released VisionOS -- VisionOS 2.
In Vision OS 2, Photos gets an upgrade that allows users to transform any 2D photo into a Spatial Photo, with added depth from moments already in their camera rolls.
Spatial Personas in the Photos app lets users view photos together, creating a more shared experience.
VisionOS 2 also supports new hand motion gestures, allowing users to access some settings more easily. For example, users can open their hands and tap to reach the home screen or turn their wrists to see the battery level.
Users who mirror their MacOS to their Vision Pro will soon get a higher resolution and bigger size, creating an ultrawide monitor view equal to two 4K monitors side by side.
The Vision Pro will also include train support for travel mode, making working during your commute easier.
Vision Pro will now feature a Guest User option that allows additional users to save their eye and hand data for 30 days.
Users can now personalize their Home View, placing apps wherever they want.
Users will now be able to watch videos in an Environment when using Safari, even on sites such as YouTube and Netflix.
Apple TV brings multiview to Vision Pro, which is especially useful when watching sports games.
Users can cast content from their iPhone, iPad, or Mac to the Vision Pro using AirPlay.
Apple is also making the Vision Pro available in more countries starting June 28. You can see the full list here.
AirPods
AirPods Pro are getting Voice Isolation to enhance call quality in noisier environments.
With the new Siri Interactions, users can now nod or shake their head "yes" or "no" when responding to Siri.
Apple is also releasing a Personalized Spatial Audio API for game developers to build around the AirPods' audio technology.
tvOS 18
Actor and music title insights on tvOS
When users watch an Apple TV+ show or movie, the new InSight feature on tvOS will include additional information such as actor names and music titles. Users can then easily add those music titles to their Apple Music playlist. When using an iPhone as a remote, the InSight information will also appear on the smartphone.
The Enhance Dialogue feature was upgraded to deliver greater vocal clarity over other elements of the movie or show, such as music or background noise, on Apple TV 4K.
Subtitles were optimized to automatically appear when the language does not match one of the devices, when users mute, or when skipping back.
Apple added support for 21:9 formatting for viewing content on projectors.
There are new, fun screensavers, including Portraits, TV and movies, and Snoopy.
When FaceTiming on tvOS 18, users will now have the option of Live Captions for English in the US and Canada.
tvOS will also feature a redesigned Apple Fitness+ experience, including new For You, Explore, and Library spaces.