AI safety rankings, one Google mega-study and a publicist’s “two bowls” theory – three things that caught my eye…
This blog post gets published every two weeks. It’s about generative AI and making content (or both at the same time). There should be no shortage of things to write about.
And yet, there are weeks when I gaze bleakly at my screen and find nothing there at all (in either my brain or on the white space in front of me).
That’s partly because I try to write less about the hype and more about the ways in which AI is useful (or not) to human beings. Which means my own human self has to be part of the idea-making process. And human selves have weeks that are itty and bitty and distracting.
Last week was one such week.
What the hell, I asked my AI-partner-in-crime Anne Taylor, can I write about? She had several suggestions, but also said: maybe you should do a post that looks at two or three things that have happened, and give your take on those things.
So, that’s what this post is about: two or three things that caught my eye over the last while.
ONE: which AI tool is the safest?
The Future of Life Institute has published its latest AI Safety Index, in which AI experts rate leading companies on key safety and security domains. There’s a 127-page PDF, for which I did not have the strength, and a two-page summary.
The various big generative AI tools are ranked along these dimensions:
- Risk assessment – this evaluates the rigour and comprehensiveness of companies’ risk identification and assessment processes for their current flagship models. The focus is on implemented assessments, not just stated commitments.
- Current harms – a focus on the AI model’s performance on safety benchmarks and the robustness of implemented safeguards against adversarial attacks.
- Safety frameworks– evaluates the companies’ published safety frameworks for frontier AI development and deployment from a risk management perspective.
- Existential safety – examines companies’ preparedness for managing extreme risks from future AI systems that could match or exceed human capabilities.
- Governance & accountability – audits whether each company’s governance structure and day-to-day operations prioritise meaningful accountability for the real-world impacts of its AI systems.
- Information sharing – evaluates how openly companies share technical, safety, and governance information, and how their public and legislative messaging align with responsible AI governance.
There’s a complex methodology where the companies are evaluated on each of these domains, and then the results are compiled into a ranking system. So, how did the big models rank, from safest to least safe? Here’s the rundown:
- Anthropic – safest
- OpenAI
- Google DeepMind
- Meta
- Z.ai
- Alibaba Cloud
The three that failed:
The report’s executive summary on the top three goes like this: “Anthropic, OpenAI, and Google DeepMind stay on top. Anthropic again earns the highest overall grade and leads five of six domains via relatively strong transparency, a comparatively established safety framework, technical research, and governance. OpenAI now leads in Risk Assessment on the strength of a broader evaluation suite and diverse engagement with external testing.”
The surprise for me was Mistral, which is European. I had assumed that meant it was subject to the extensive European legal structure. But the report recommends that to improve its dismal grade, it needs to publish a full safety framework and governance structure, engage substantively with existential safety (leadership is said to consistently downplay – and at times dismiss – frontier risk rather than articulating any control or alignment strategy) and improve weak safety benchmark performance.
Overall though, the three top tools are the ones I use, and the ones I recommend. One thing: being the safest company on a list like this doesn’t mean actual safety; it means being safe compared to other offerings. They all have privacy issues; they are all companies that need watching and critical thought.
TWO: how are people using AI at work?
Google’s crunched the numbers: Understanding the AI economy. Those numbers are very, very big (this is Google, how could it be otherwise?)
Their study (called the AI & Economy ATLAS, for Activity, Task, Landscape, and Adoption Study) used a dataset built from 15 million aggregated and de-identified human-AI interactions. That means more than 1 billion people monthly, in over 150 countries, speaking 140 languages, in 800 occupations, and doing 4,000 tasks.
So what do we learn from this? Here are the findings, as reported by Google, and much shortened by me:
- AI use at work is broad but shallow: Workplace adoption spans all industry sectors and also 68% of all occupations. But people are using AI selectively: in a typical job AI is used for only ~21% of tasks.
- At work, most AI use is focused on collaboration and assistance with tasks: The vast majority of AI interactions at work focus on collaborative uses such as ideation, strategy, information retrieval, and learning.
- AI use is not limited to white collar workers: Workers in manual and technical trades (e.g., auto technicians, industrial mechanics) are using conversational AI as a live collaborator for real-time diagnostics, troubleshooting, and on-the-fly learning.
- AI is delivering value at home: Over 86% of interactions with AI tools in ATLAS occur outside of work. People are using AI for things like researching purchases, help with using appliances and navigating government services like taxes, licensing, and fines.
- Global AI adoption is tracking GDP per capita: The data shows AI usage in over 150 countries and territories that represent 99% of the world’s population. English represents only about a third of global AI conversations, and users do not systematically abandon their native languages for complex tasks. “On a per-capita basis AI usage closely mirrors a country’s relative level of wealth, raising concerns about a persisting digital divide. However this isn’t a universal rule: some middle-income countries in South America and the Middle East are adopting AI at rates comparable to higher-income countries.”
What do I make of all this? I have one caveat. Google says that one source of the data was its AI Mode – that stuff that appears when you type a query in their search engine. Since none of us have much choice about that, I don’t know that a lot can be inferred about whether people actually intended to use AI to find out something they wanted to know. And it might also skew the results on languages used: people will be using phones set to their own languages, and seeing AI mode in those languages (I assume).
Otherwise, Google’s vast research doesn’t really deliver any surprises. The training Safe Hands has done pretty much duplicates these patterns: more people use AI than any of us think, and they use it for a wide range of purposes, but it ain’t changing lives in any deep way. The cheering surprise for me was the extent to which English is not the dominant language in AI use; that’s something that’s hard to see from the small set of people we have worked with.
THREE: existential questions
I encountered book publicist Fauzia Burke at an AI-in-publishing online webinar I attended and was entranced by her “two bowls” take on when to use generative AI, and when not to use it.
You can read the whole thing here but the essence of it is this: Imagine you have two bowls in your kitchen. One is the utilitarian, everyday stainless steel bowl. “If a tiny bit of eggshell falls in, it’s not the end of the world. This bowl is for your repetitive, administrative tasks, the tasks that cause burnout. AI is perfect for these tasks.” Then there’s the handmade ceramic bowl, which comes out for special occasions. “This bowl is for the things that should remain human: your relationships, your writing, your creative intuition, and your unique taste.”
So simple, so beautiful. I signed up on her Substack page instantly, where her most recent post brought a question I think about a lot.
At a conference where Burke was a speaker, a young woman said this from the floor: “I don’t understand why we’re talking about AI like it’s inevitable. I don’t use it. None of my friends use it. And we don’t plan to.”
But when Burke asked the room for a show of hands from people who use AI every day, most hands in the room went up. Says Burke:
This explains the tension we’re all sitting in right now. Because both things are true at the same time: some people are not using AI at all, while others use it constantly.
Burke’s take on this tension? “AI adoption isn’t moving in a single, clean direction. We tend to talk about it like a wave that everyone is either ahead of or behind. But it doesn’t feel like that on the ground.” She uses AI because it makes her life easier and frees her up to do the things only she can do.
She theorises that when someone asks, “Why are we acting like this is inevitable?” what they might really be asking is: “Why does it feel like a decision has already been made for me?”
No easy answers here – but I do think many of the in-the-shadows feelings about generative AI come down to questions of choice and agency: so much of our 21st century life is beyond our control that adding AI to the mix feels like a bridge too far for many people.
It was that feeling that paradoxically that led me down the “learn about AI” path. If it was going to change my working life in ways I could not control, I at least wanted to understand what was happening. Which led to this series of blog posts. And that episodic feeling that I don’t have anything to write about. Turns out I was wrong!
Main picture: Compagnons, Unsplash
Previous Sensible Guide articles
Navigating AI hype: Five newsletters that will help – Everywhere you look, there’s an article about artificial intelligence. Here’s a list of the people I follow to help in navigating AI hype.
Which AI tool to use? Start by playing – Knowing which AI tool to use can be hard to figure out – there are apparently 10,500 tools out there. I suggest a place to start…
AI and humanity – a mission statement – Three things I think about how we humans can use AI to enhance our lives.
If any of this is relevant to your work…
You might find one of these useful: