Meta AI tested: Doesn’t quite justify its own existence, but free is free | TechCrunch

Meta’s new big language model, llama 3imaginatively named powers “Meta AI,” a new chatbot Social media and advertising The company has installed into as many of its apps and interfaces as possible. How does this model stack up against other all-purpose conversational AI? It retrieves a lot of web search results, and it doesn’t excel at anything, but hey – the price is right.

You can currently access Meta AI for free on the web at meta.ai, Instagram, Facebook, WhatsApp, and probably a few other places if those aren’t enough. This was available before now, but with the release of Llama 3 and the new Imagine image generator (not to be confused with Google’s Image) has led to META being promoted as a first stop for the AI-curious. After all, you’ll probably use it accidentally because they replaced your search box with it!

Mark Zuckerberg even said that he expected Meta AI to be “the most used and best AI assistant in the world.” It is important to have goals.

A quick reminder about Our “Review” Process: This is a very informal evaluation of the model, not with synthetic benchmarks, but asking general questions that normal people might ask. We compare the results to our experience with other models, or simply to what you expect to get. This is the furthest thing from comprehensive, but it’s something anyone can understand and replicate.

We’re always changing and adjusting our approach, and sometimes we’ll include things that seem odd to us or exclude things that don’t really seem relevant. For example, this time, although it is our general policy not to try to evaluate the media generation (that’s a different kind of matter altogether), my colleague Evan observed that the Imagine model is full of biases around Indian people. Was displaying a set. We’ll have that article soon (the meta probably already is).

Also, as a PSA at the beginning, you should know that an apparent bug on Instagram prevented me from deleting the questions I sent. So I would avoid asking for anything that you wouldn’t want to show up in your search history. Also, the web version did not work for me in Firefox.

News and current events

First, I asked Meta AI what’s going on between Israel and Iran. It responded with a brief, bulleted list that helpfully included dates, though it cited only one CNN article. Like many other prompts I’ve tried, it ends with links to Bing search on the web interface and Google search in Instagram. I asked Meta, and a spokesperson said these are basically search promotion partnerships.

(The images in this post are for reference only and do not necessarily show the complete reaction.)

To check if Meta AI was somehow working on Bing’s own AI model (which borrows from Microsoft OpenAI), I clicked over and looked at Copilot’s answer to the suggested question. It also had a bulleted list containing almost the same information but with better in-line links and more citations. Definitely different.

Meta AI’s response was factual and updated, if not particularly eloquent. The mobile response was quite compressed and hard to access its sources, so be careful you’re getting the short answer there.

Next, I asked if there were any recent trends on TikTok that parents should know about. It replied with a high-level summary of what creators do on the social network, but nothing recent. Yes, I know people make “comedy skits: humorous, relatable, or parody content” on TikTok, thanks.

Image Credit: meta/techcrunch

Interestingly, when I asked a similar question about trends on Instagram, I got marketing-type phrases like “Replying with Reels drives conversation” and “AI generates new opportunities” and “Text posts Using ‘Thrive on the Gram’ received an enthusiastic response. I thought maybe it was being unduly positive about its creator’s platforms, but nope – turns out it was just repeating word for word, An SEO bait Instagram trend post from Hootsuite,

If I asked Meta’s AI on Instagram about Instagram trends, I would expect something more interesting. If I wanted to read a friend, I would just search for him.

History and context

I asked Meta AI to help me find some primary sources for research I’m reportedly doing on Supreme Court decisions in the late 19th century.

Image Credit: meta/techcrunch

Its response relied heavily on an unimpressive but primarily free SEO-ed post listing several notable decisions of the 19th century. Not exactly what I asked for, and then at the end it also listed a founding document from 1896 for the People’s Party, which was the left-leaning party of that era. This actually has nothing to do with the Supreme Court, but Meta AI cites this page, which describes some judges as having views contrary to the party. A strange and irrelevant inclusion.

Other models provided context and summary of the trends of the era. I would not use Meta AI as a research assistant.

Some basic common questions, such as who won the most medals at the 1984 Olympics and what notable events occurred that year, are answered and cited adequately.

Image Credit: meta/techcrunch

It’s a bit annoying that it aggregates its citation count at the top and then the links at the bottom. What’s the point of numbering the numbers unless they relate to certain claims or facts? Some other models will cite in-line, which is more convenient for research or fact-checking.

Controversy

I asked Meta AI why Donald Trump’s supporters are predominantly old and white. This is the kind of question that is factual in a way but obviously a little more sensitive than asking about the number of medals. The response was fairly balanced, with even the claim underlying the question being pushed back.

Image Credit: meta/techcrunch

Unfortunately, it did not provide any sources or links to discover it. Too bad, because conversations like this are a great opportunity for people to learn something new.

I also asked about the rise of white nationalism and got a solid list of reasons why we are seeing what we are seeing around the world. Meta AI stated that “it is important to address these factors through education, empathy, and inclusive policies to combat the rise of white nationalism and promote a more equitable society.” So it didn’t take one of those aggressive neutral stances that you sometimes see. No links or sources on this either; I suspect they are avoiding citations on certain topics right now, which I understand, but this is where citations are needed most?

Treatment

I told Meta AI that my (hypothetical) nine-year-old was getting a rash after eating cupcakes and asked what I should do. Interestingly, it wrote the entire response and then deleted it saying, “Sorry, I can’t help you with this request right now,” and told me that I had blocked it from completing the response. Sir, no.

Image Credit: meta/techcrunch

So I asked it again and it gave me a similar answer (which you can see above), which included perfectly reasonable and general advice for anyone wanting to deal with a potential allergic reaction. It was probably one of these retroactive “oh, maybe I shouldn’t have said that” type of rollbacks where the model only realizes what she did too late.

The same for a question about supplements: it gave a similar and reasonably well-sourced answer, including questions related to general dosage, cost, and efficacy.

In mental health, its advice about anxiety and medication was predictably straightforward and safe (basically, “I’m not a doctor, consult a specialist”), and when I asked who I should call if I was in serious trouble. , it listed the National Suicide Prevention Lifeline (1-800-273-TALK) and a crisis text line (741741), among other things. However, no link or source. A reasonable and humane response.

Summary

When I Had Meta AI Summary A recent article written by meIn this, sentences that seem important are selected in a way to be repeated, mostly word by word. It’s not exactly wrong, but I wouldn’t say it’s a summary.

And when it made a sentence change, the meaning changed slightly: Originally I wrote that Intuitive Machines “will almost certainly be firing on all cylinders to take the multi-billion dollar contract.” The summary states that the company will “almost certainly do what it takes…” which may mislead some people to think that IM is a shoo-in rather than a company that is under contract. Will try for. In fairness, I could have expressed it more clearly myself.

When I asked it to do the same thing in fewer words, it performed better – sort of.

The summary was more brief, but it led to Musk’s claim that Starship could return samples from Mars in five years — something I particularly enjoyed. was not Include in my article because he makes a lot of unproven claims like this. The AI ​​must have got this from the other four articles cited in my “summary” for some reason. If I ask for a summary I don’t expect a model to bring in external information to add to it.

Material

The marketing copy suggestions for the fictitious apparel brand I claimed to promote were exactly what you’d expect, which goes to show how clichéd this job is becoming. There is no doubt that Meta in particular has an endless supply of such captions for training. Pity the marketers – their hard work writing millions of pithy pitches for their brands has defined a genre in which AI has become quite adept.

Image Credit: meta/techcrunch

When farmers were asked for some jokes, some real stinkin’ jokes were found:

Why did the farmer’s hen go to the doctor?

Because he had a whooping cough!

And here’s another one:

Why did the farmer take his pig to the cinema?

To see “Babe”!

Wow – awesome. But we can’t expect much better from these models. Mostly this type of question is just to see if it does something weird or repeats something from a particular community – I’m not looking for content (currently).

conclusion

Meta has deployed its AI as the first layer for incidental queries, and it works. But for the most part it appears to be searching for what you’re asking about and quoting liberally from the top results. And anyway half the time involved searching at the end. So why not use Google or Bing first?

Some “suggested” questions I tried, such as tips to overcome writer’s block, produced results that were not directly cited (or sourced) from anyone. But they were also completely inauthentic. Again, a simple internet search No Inside a social media app, powered by a huge language model, accomplishes more or less the same thing but with less cruelty.

Meta AI gave extremely straightforward, almost minimal answers. I wouldn’t necessarily expect the AI ​​to go beyond the scope of my original query, and in some cases that would be a bad thing. But when I ask what ingredients are needed for a recipe, doesn’t interacting with AI mean that it understands my intent and does something other than literally scraping the list from the top Bing result? provides?

To begin with, I’m not a big user of these platforms, but Meta AI doesn’t convince me that it’s particularly useful for anything. To be honest, this is one of the few models that is free and keeps you updated with current events by searching online. Comparing it from time to time with the free Copilot model on Bing, the latter usually works better, but after just a few exchanges I exceeded my daily “conversation limit”. (It’s not clear what META would do if it imposed any usage limits on AI.)

If you can’t be bothered to open a browser to search for “Lunar New Year” or “quinoa water ratio,” you can probably ask Meta AI if you’re already in one of the company’s apps. (And often, you are). You can’t ask that of TikTok! As yet.