> previewed “as early as December,” ... After that, Jarvis might be made available to early testers, so a launch does not seem imminent.
Google is trying to show they are not behind in the AI race by advertising something probably barely out of alpha testing. It just reinforce the idea that Gemini is still inferior to Claude and ChatGPT.
I tried Gemini once and then tried Claude. It was such a huge difference I can't imagine Google, who created the transformer architecture, can be so behind a tiny startup a fraction of their size.
I flip between ChatGPT4o, Claude, Gemini (and its offsets NotebookLM and AIStudio). They all have their niches for example Claude projects (+ 3rd party claudesync) is useful for generating code for an Xcode project, or AIStudio can handle more file formats including video. Then there's different context sizing, Gemini being the biggest I'm aware of.
I'm really unimpressed by the velocity of feature development from these AI orgs, I don't expect them to have complete feature parity any time soon if at all.
As always pick the right tool for the job. There is almost never a 1-size fits all best selection.
> I'm really unimpressed by the velocity of feature development from these AI orgs, I don't expect them to have complete feature parity any time soon if at all.
I feel two ways about this: one is that there's a lot of opportunity for fast moving startups. But two: how does a startup remain defensible with the giant comes along?
The example is copilot: Microsoft announced it 19 months ago and still very rough around the edges and many opensource projects are doing a fairly decent job of filling in the gap in the meanwhile.
There's a huge difference between Gemini and Gemini Pro.
> I'm really unimpressed by the velocity of feature development from these AI orgs
Seems about right though. Do they even know what to build? I get the sense that they are getting an idea started and then moving mostly based on feedback. We're still very early in this game.
This is what I wanted to know. Unfortunately, they don't allow you to preview the pro for free and it's hard to pay for something that you have no idea if it's good.
I have found Gemini pro to be excellent for RAG application. Most of my java projects with less than 500k lines of code can fit entirely in its context window so I don't have to mess with chunking.
Before this came along, we had tried different tools and RAG applications and nothing compares to what Gemeni delivers. And the cost is nearly nothing compared to gains.
Gemini Flash seems remarkably fast and cheap, noticeably cheaper than most (any?) reasonable alternatives. Other models have a best in class context window. Gemini is also known for citing its sources better than many other models.
I’m not sure they’re behind, maybe just focusing on different things? Being fast makes sense for a lot of use cases, and large context windows are important for the sorts of cases like NotebookLM, and citing sources is important for safety.
Gemini is mostly good for its large context window (which is a huge plus), but the answers and “intelligence” aren’t nearly as good as chatGPT 4o. I haven’t tried Claude so I don’t know how well that does.
I have not been able to find an easy way to use Claude. While ChatGPT lets me use without even logging in (for free), the only way I could find for Claude was to add an email - approve it by clicking a link on it - and be presented with a mobile number verification.
But ChatGPT - hallucinates and flounders so much about almost anything worthwhile I ask that it is simply of no worth to me as far as trustworthiness is concerned. It tries to be flowery to give an impression of being "good". It is not. Is it decent fort writing quick office replies which you could quick-edit and sent. I would think so. Anything more "serious"? Nope!
Gemini (could use it via a non-Gmail throwaway email Google a/c) wasn't that verbose or going all over. It was more restrained and didn't try too hard about things it didn't know or couldn't do anything about.
I think a lot of the reason ChatGPT seems "better" is because it is easily accessible and the company/founder actually achieved the intended "viral marketing" it could including by that firing and re-hiring saga, ScaJo episode et al.
The endgame is undisclosed promotional content seamlessly inserted into algorithmic output. And of course this will be too indirect and obfuscated to be regulated. If you think lack of corporate accountability is bad now, wait until all reputation and liability have been fully laundered to AI.
Couldn't regulators simply target the other end? As in, denying businesses the right to "buy AI generated ad space"? Or denying ad companies a license to include seamless ads in AI generated output?
Billboards on the highway are limited in scope due to safety and other reasons. A billboard can't have mechanical arms that swing about causing driver distraction. If "information safety" is becoming a thing, the equivalent of "no mechanical arms on billboards" might be enforced on AI generators? Or am I suggesting a remedy worse than the problem is solves?
Consider the various restrictions on tobacco advertising. Now think about how newly legal gambling advertises. What regulations we have are largely a vestige of a different time. Half the politicians in this era put business over people, the other half don't have the legislative power to put people over business.
I suppose... If I ask Google's AI to walk me through the process of baking a chocolate cake, I don't care if the AI seamlessly recommends a brand of flour or appliance. As long as it doesn't compromise the fundamental cake-making advice.
Where it gets problematic, is if the AI pushes a specific brand as a necessary step of the cake-making process. Suddenly it's unethical.
It would also be a problem if the AI recommended a competing brand of appliance if I were specifically asking the AI to tell me how to use XYZ brand of appliance. Kind of like how Google lets advertisers buy ads for competitor keywords, which in my opinion is grubby and borderline unethical.
Google Gemini can't even set reminders on my Pixel 9 Pro. It's really hard for me to get excited about Google AI products when it can't even match the functionality of their non-ai products that they're supposed to replace.
Just tried it. "Remind me tomorrow to get energy drinks at Costco" returns "Sorry, I can't help with that" and instructions for creating a reminder.
Edit: it looks like I need to turn "app activity" on, which means I need to opt in to someone being allowed to read my interactions and annotate them, making them undeletable. Then I need to connect Google Workspace. Then it creates a reminder but tapping on it prompts me to install the Google Tasks app. It's absolutely clown shoes for something that could be done on the device with existing APIs.
At the moment none of these tools are sticky; switching is easy; trial costs are low.
This feels more like a sticky move than an innovation in what AI can do.
This is the know more about you, better context for results trick. Which any other could deliver with your "user habit profile RAG data" - browser hook is a great way to collect.
I’ve used the paid version of Gemini and was underwhelmed with its limitations. The integrations in workspace seem like they barely tried. It can’t even recognize spreadsheet headers to answer basic questions, despite having a private level of access to your docs that exceeds what a third party could. Is this chrome AI going to be any better, or just a way for them to implement the collection of training data as quickly as possible to avoid irrelevance?
It really says something about the state of competition, the power of capital, and the level of data hoarding these megacorps enjoy. A startup fumbling this way would be dead on arrival and receive no second chances.
There’s also something very offensive about Google championing the death of ad blockers in browsers while sucking in all our data to power their invasive browser features.
Gemini is terrible. It's way worse than even GPT 3. Never mind 3.5 or Claude. It's basically useless. Even the simplest things like trivial code transformations don't work. Gemini goes rogue all the time and starts to do things it shouldn't.
I get the feeling that in desperation people at Google are hacking the metrics to make their model look good. While in reality it's just junk.
No model, and I've tried a lot of them, has such a massive gap between good benchmark performance and horrible real world performance.
Honestly this sounds like you tried one unrepresentative query at launch and nothing else. I’m using Gemini regularly now and it’s not bad. Via the API I’ve had even more success. Much better than 3.5, and competitive with 4 in the things I’ve used it for, plus a huge context window.
I agree with you. For business applications, context window is more important than any extra reasoning ability. I can load up nearly all business documents, regulations and rules and ask it questions and the responses are accurate and better that models with small context window.
If you knew this ahead of time then what value did it provide you? Put another way, once the counter party realizes that you've pushed this responsibility onto an LLM, aren't you worried that they could take advantage of this fact to produce intentionally misleading query results?
In an adversarial world what long term value could possibly exist here?
>> If you knew this ahead of time then what value did it provide you?
I think I should have clarified that the responses were accurate when we bench-marked them against RAG applications built on other LLMs. If the results had been poor or below a set threshold, the RAG application built on Gemini Pro would have been shelved.
>> Put another way, once the counter party realizes that you've pushed this responsibility onto an LLM, aren't you worried that they could take advantage of this fact to produce intentionally misleading query results?
This is an excellent point but inapplicable in our use case as this was only meant for internal users, who otherwise would be opening word and PDF documents and searching through them or relying on their own memory.
> previewed “as early as December,” ... After that, Jarvis might be made available to early testers, so a launch does not seem imminent.
Google is trying to show they are not behind in the AI race by advertising something probably barely out of alpha testing. It just reinforce the idea that Gemini is still inferior to Claude and ChatGPT.
I tried Gemini once and then tried Claude. It was such a huge difference I can't imagine Google, who created the transformer architecture, can be so behind a tiny startup a fraction of their size.
I flip between ChatGPT4o, Claude, Gemini (and its offsets NotebookLM and AIStudio). They all have their niches for example Claude projects (+ 3rd party claudesync) is useful for generating code for an Xcode project, or AIStudio can handle more file formats including video. Then there's different context sizing, Gemini being the biggest I'm aware of.
I'm really unimpressed by the velocity of feature development from these AI orgs, I don't expect them to have complete feature parity any time soon if at all.
As always pick the right tool for the job. There is almost never a 1-size fits all best selection.
> I'm really unimpressed by the velocity of feature development from these AI orgs, I don't expect them to have complete feature parity any time soon if at all.
I feel two ways about this: one is that there's a lot of opportunity for fast moving startups. But two: how does a startup remain defensible with the giant comes along?
The example is copilot: Microsoft announced it 19 months ago and still very rough around the edges and many opensource projects are doing a fairly decent job of filling in the gap in the meanwhile.
Are fast moving startups the best vehicle for rapid software development in 2024?
There's a huge difference between Gemini and Gemini Pro.
> I'm really unimpressed by the velocity of feature development from these AI orgs
Seems about right though. Do they even know what to build? I get the sense that they are getting an idea started and then moving mostly based on feedback. We're still very early in this game.
This is what I wanted to know. Unfortunately, they don't allow you to preview the pro for free and it's hard to pay for something that you have no idea if it's good.
I have found Gemini pro to be excellent for RAG application. Most of my java projects with less than 500k lines of code can fit entirely in its context window so I don't have to mess with chunking.
Before this came along, we had tried different tools and RAG applications and nothing compares to what Gemeni delivers. And the cost is nearly nothing compared to gains.
Gemini Flash seems remarkably fast and cheap, noticeably cheaper than most (any?) reasonable alternatives. Other models have a best in class context window. Gemini is also known for citing its sources better than many other models.
I’m not sure they’re behind, maybe just focusing on different things? Being fast makes sense for a lot of use cases, and large context windows are important for the sorts of cases like NotebookLM, and citing sources is important for safety.
Gemini is #3 on the leaderboard; seems like it can't be that bad.
https://huggingface.co/spaces/lmarena-ai/chatbot-arena-leade...
I worked on this at Google 4 years ago
You might have been recycling what I did at Google 8 years ago.
(Friendly reminder not to trust everything you read on the internet)
Gemini is mostly good for its large context window (which is a huge plus), but the answers and “intelligence” aren’t nearly as good as chatGPT 4o. I haven’t tried Claude so I don’t know how well that does.
I have not been able to find an easy way to use Claude. While ChatGPT lets me use without even logging in (for free), the only way I could find for Claude was to add an email - approve it by clicking a link on it - and be presented with a mobile number verification.
But ChatGPT - hallucinates and flounders so much about almost anything worthwhile I ask that it is simply of no worth to me as far as trustworthiness is concerned. It tries to be flowery to give an impression of being "good". It is not. Is it decent fort writing quick office replies which you could quick-edit and sent. I would think so. Anything more "serious"? Nope!
Gemini (could use it via a non-Gmail throwaway email Google a/c) wasn't that verbose or going all over. It was more restrained and didn't try too hard about things it didn't know or couldn't do anything about.
I think a lot of the reason ChatGPT seems "better" is because it is easily accessible and the company/founder actually achieved the intended "viral marketing" it could including by that firing and re-hiring saga, ScaJo episode et al.
How many ads-per-second will Jarvis serve?
In all seriousness I firmly believe they’ll embrace ads in AI responses and I see zero reason to think they wouldn’t.
The endgame is undisclosed promotional content seamlessly inserted into algorithmic output. And of course this will be too indirect and obfuscated to be regulated. If you think lack of corporate accountability is bad now, wait until all reputation and liability have been fully laundered to AI.
Couldn't regulators simply target the other end? As in, denying businesses the right to "buy AI generated ad space"? Or denying ad companies a license to include seamless ads in AI generated output?
Billboards on the highway are limited in scope due to safety and other reasons. A billboard can't have mechanical arms that swing about causing driver distraction. If "information safety" is becoming a thing, the equivalent of "no mechanical arms on billboards" might be enforced on AI generators? Or am I suggesting a remedy worse than the problem is solves?
Consider the various restrictions on tobacco advertising. Now think about how newly legal gambling advertises. What regulations we have are largely a vestige of a different time. Half the politicians in this era put business over people, the other half don't have the legislative power to put people over business.
I suppose... If I ask Google's AI to walk me through the process of baking a chocolate cake, I don't care if the AI seamlessly recommends a brand of flour or appliance. As long as it doesn't compromise the fundamental cake-making advice.
Where it gets problematic, is if the AI pushes a specific brand as a necessary step of the cake-making process. Suddenly it's unethical.
It would also be a problem if the AI recommended a competing brand of appliance if I were specifically asking the AI to tell me how to use XYZ brand of appliance. Kind of like how Google lets advertisers buy ads for competitor keywords, which in my opinion is grubby and borderline unethical.
Google Gemini can't even set reminders on my Pixel 9 Pro. It's really hard for me to get excited about Google AI products when it can't even match the functionality of their non-ai products that they're supposed to replace.
You should double check that you can't do this. I just created a reminder from the Gemini app on Android.
Just tried it. "Remind me tomorrow to get energy drinks at Costco" returns "Sorry, I can't help with that" and instructions for creating a reminder.
Edit: it looks like I need to turn "app activity" on, which means I need to opt in to someone being allowed to read my interactions and annotate them, making them undeletable. Then I need to connect Google Workspace. Then it creates a reminder but tapping on it prompts me to install the Google Tasks app. It's absolutely clown shoes for something that could be done on the device with existing APIs.
Takes screenshots and uploads them to the cloud for processing because it is so inefficient?
Google’s not even pretending to care about privacy any more.
At the moment none of these tools are sticky; switching is easy; trial costs are low.
This feels more like a sticky move than an innovation in what AI can do.
This is the know more about you, better context for results trick. Which any other could deliver with your "user habit profile RAG data" - browser hook is a great way to collect.
With ChatGPT integration with iOS 18.2, the stickiness is starting to appear
I’ve used the paid version of Gemini and was underwhelmed with its limitations. The integrations in workspace seem like they barely tried. It can’t even recognize spreadsheet headers to answer basic questions, despite having a private level of access to your docs that exceeds what a third party could. Is this chrome AI going to be any better, or just a way for them to implement the collection of training data as quickly as possible to avoid irrelevance?
It really says something about the state of competition, the power of capital, and the level of data hoarding these megacorps enjoy. A startup fumbling this way would be dead on arrival and receive no second chances.
There’s also something very offensive about Google championing the death of ad blockers in browsers while sucking in all our data to power their invasive browser features.
No thanks
What is Google doing?
Gemini is terrible. It's way worse than even GPT 3. Never mind 3.5 or Claude. It's basically useless. Even the simplest things like trivial code transformations don't work. Gemini goes rogue all the time and starts to do things it shouldn't.
I get the feeling that in desperation people at Google are hacking the metrics to make their model look good. While in reality it's just junk.
No model, and I've tried a lot of them, has such a massive gap between good benchmark performance and horrible real world performance.
Honestly this sounds like you tried one unrepresentative query at launch and nothing else. I’m using Gemini regularly now and it’s not bad. Via the API I’ve had even more success. Much better than 3.5, and competitive with 4 in the things I’ve used it for, plus a huge context window.
I agree with you. For business applications, context window is more important than any extra reasoning ability. I can load up nearly all business documents, regulations and rules and ask it questions and the responses are accurate and better that models with small context window.
> and the responses are accurate
If you knew this ahead of time then what value did it provide you? Put another way, once the counter party realizes that you've pushed this responsibility onto an LLM, aren't you worried that they could take advantage of this fact to produce intentionally misleading query results?
In an adversarial world what long term value could possibly exist here?
>> If you knew this ahead of time then what value did it provide you?
I think I should have clarified that the responses were accurate when we bench-marked them against RAG applications built on other LLMs. If the results had been poor or below a set threshold, the RAG application built on Gemini Pro would have been shelved.
>> Put another way, once the counter party realizes that you've pushed this responsibility onto an LLM, aren't you worried that they could take advantage of this fact to produce intentionally misleading query results?
This is an excellent point but inapplicable in our use case as this was only meant for internal users, who otherwise would be opening word and PDF documents and searching through them or relying on their own memory.
How can I stop this bullshit from ever affecting my life?
https://getfirefox.com
You can't.
FAANGs don't understand the concept of consent, let alone informed consent.
Turn off the computer
[dead]