At I/O 2024, Google announced a great new AI feature for Google Photos, simply called Ask Photos. With Ask Photos, you can treat the app like a chatbot, say, Gemini or ChatGPT: You can request a specific photo in your library, or ask the app a general question about your photos, and the AI will sift through your entire library to both find the photos and the answers to your queries.

How does Ask Photos work?

When you ask Ask Photos a question, the bot will make a detailed search of your library on your behalf: It first identifies relevant keywords in your query, such as locations, people, and dates, as well as longer phrases, such as "summer hike in Maine."

After that, Ask Photos will study the search results, and decide which ones are most relevant to your original query. Gemini's multimodal abilities allow it to process the elements of each photo, including text, subjects, and action, which helps it decide whether that image is pertinent to the search. Once Ask Photos picks the relevant photos and videos for your query, it combines them into a helpful response.

Google says your personal data in Google Photos is never used for ads and human reviewers won't see the conversions and personal data in Ask Photos, except, "in rare cases to address abuse or harm." The company also said they don't train their other AI products with this Google Photos data, including other Gemini models and services.

What can you do with Ask Photos?

Of course, Ask Photos is an ideal way to quickly find specific photos you're looking for. You could ask, "Show me the best photos from my trip to Spain last year,” and Google Photos will pull up all your photos from that vacation, along with a text summary of its results. You can use the feature to arrange these photos in a new album, or generate captions for a social media post.

However, the more interesting use here is for finding answers to questions contained in your photos without having to scroll through those photos yourself. Google shared a great example during its presentation: If you ask the app, "What is my license plate number?" it will identify your car out of all the photos of cars in your library. It will not only return a picture of your car with your license plate, but will answer the original question itself. If you're offering advice to a friend about the best restaurants to try in a city you've been to, you could ask, "What restaurants did we go to in New York last year?" and Ask Photos will return both the images of the restaurants in your library, as well as a list you can share.

When will Ask Photos be available?

Google says the experimental feature is rolling out in the coming months, but no specific timeframe was given.

AI has been the dominating force in this year's Google I/O—and one of the biggest announcements that Google made was a new Gemini model customized for Google Search. Over the next few weeks, Google will be rolling out a few AI features in Search, including AI Overviews, AI-organized search results, and search with video.

AI Overviews

When you're searching for something on Google and want a quick answer, AI Overviews come into play. The feature gives you an AI-generated overview of the topic you're searching for, and cites its sources with links you can click through for further reading. Google was testing AI Overviews in Search Labs, but has been rolling out the feature to everyone in the U.S. this week.

At a later date, you'll be able to adjust your AI Overview with options to simplify some of the terminology used, and even break down results in more detail. Ideally, you could turn a complex search into something accessible for anyone. Google is also pushing a feature that lets you stack multiple queries into one search: The company used the example of “find the best yoga or pilates studios in Boston and show me details on their intro offers, and walking time from Beacon Hill," and AI Overviews returned a complete result.

As with other AI-generated models, you can also use this feature to put together plans of action, including creating meal plans and prepping for a trip.

AI-organized results

In addition to AI Overviews, Google Search will soon be using generative AI to create an "AI-organized results page." The idea is the AI will intelligently sort your most relevant options for you, so you won't have to do as much digging around the web. So when you're searching for something like, say, restaurants for a birthday dinner, Google's AI will suggest the best options it can find, organized beneath AI-generated headlines. AI-organized results will be available for English searches in the U.S.

Search with video

Google previously rolled out Circle to Search, which lets you circle elements of your screen to start a Google search for that particular subject. But soon, you'll also be able to start a Google search with video. The company gave an example of a customer who bought a used record player whose needle wasn't working properly. The customer took a video of the issue, describing it out loud, and sent it along as a Google search. Google analyzed the issue and returned a relevant result, as if the user had simply typed out the problem in detail.

Search with video will soon be available for Search Labs users in English in the U.S. Google will expand the feature to more users in the coming months.

At I/O 2024, Google made lots of exciting AI announcements—but one that has everyone talking is Project Astra. Essentially, Project Astra is what Google is calling an "advanced seeing and talking responsive agent." This means that a future Google AI will be able to get context from what's around you and you can ask a question and get a response in real time. It's almost like an amped-up version of Google Lens.

Project Astra is being developed by Google's DeepMind team, which is on a mission to build AI that can responsibly benefit humanity; this project is just one of the ways it's doing so. Google says that Project Astra is built upon its Gemini 1.5 Pro, which has gained improvements in areas such as translation, coding, reasoning and more. As part of this project, Google says they've developed prototype AI agents that can process information even faster by continuously encoding video frames and combining video and speech input into a timeline of events. The company is also using their speech models to enhance how its AI agents sound, for a wider range of intonations.

Google released a two-part demo video to show off how Project Astra works. The first half of the video shows Project Astra running on a Google Pixel phone; the latter half shows the new AI running on a prototype glasses device.

In the demo video, we can see the user using their Pixel phone with a camera viewfinder open and moving their device around the room while asking the next-generation Gemini AI assistant, "Tell me when you see something that makes sound" and the AI responding by pointing out the speaker on the desk. Other examples in the video include asking what a part of the code on a computer screen does, what city neighborhood they're currently in and coming up with a band name for a dog and its toy tiger.

While it will be a long time before we see this next-generation AI from Project Astra coming to our daily lives, it's still quite cool to see what the future holds.

As part of Google I/O, the company made several AI announcements, including a new experimental tool called VideoFX. With VideoFX, users can have a high-quality video generated just by typing in a prompt—and it's powered by a new AI model that Google calls Veo.

What is Google Veo?

To get the new VideoFX tool to work, Google's DeepMind team developed a new AI model called Veo. The AI model was specifically created with video generation in mind and has a deep understanding of natural language and visual semantics, meaning you could give it prompts such as "Drone shot along the Hawaii jungle coastline, sunny day" or "Alpacas wearing knit wool sweaters, graffiti background, sunglasses."

The Veo model is clearly Google's answer to OpenAI's Sora AI video generator. While both Veo and Sora can create realistic videos using AI, Sora's videos had a limit of 60 seconds. Meanwhile, Google says that Veo can generate 1080p videos that can go beyond a minute in length, although Google doesn't specify how much longer.

What is VideoFX?

Essentially, the new VideoFX tool takes the power of the new Veo AI model and puts it into an easy-to-use video editing tool. You'll be able to write up your prompts and they'll be turned into a video clip. The tool will also feature a Storyboard mode that you can use to create different shots, add music, and export your final video.

Google's VideoFX tool is currently being tested in a private preview in the U.S. The company hasn't said when the VideoFX tool will be publicly available beyond this preview.

How to sign up for VideoFX?

For those interested in trying the new Veo-powered tools in VideoFX, you can join the waitlist: Go to http://labs.google/trustedtester and submit your information. Google will be reviewing all submissions on a rolling basis.