What is “the algorithm?”
Recommendation algorithms are everywhere. They constantly suggest us new music, TV shows, dinner, and even potential romantic partners — and for the most part they can be pretty good at predicting what we like. How?
Broadly, algorithms are a specific set of instructions that when followed should always result in a desired outcome, whether that’s solving a problem or performing a task. Although they’ve been around for thousands of years, computers and the internet have made it easier for complex algorithms to be performed almost instantly. Algorithms used specifically to make recommendations work by using data from past behavior to predict future behavior.
To get a better sense of how that works, we have to go back to 2006, when over 30,000 people around the world came together to improve one specific algorithm. More on that below.
Origin story
The Netflix Prize
Back when it was just a DVD rental service, Netflix launched a competition known as the Netflix Prize. The company offered a $1 million reward to anyone who could increase the success rate of its movie recommendation software “CineMatch” at predicting what movies their users would like by 10%. As part of the competition, the company publicly shared data that included 100 million ratings of 17,770 movies from 480,189 customers.
Through online forums, the various competitors shared their progress and ideas on how to improve their algorithms. Participants discussed things like whether the time of day someone rated a movie matter and how likely it was that someone would enjoy a sequel if they liked the original.
One competitor was particularly influential. He went by the pseudonym Simon Funk. He came up with the idea of using a math technique called singular value decomposition (SVD) to sort Netflix’s data to automatically find similarities among movies users liked, such as genres and the actors starring in a film. These factors, identified by the algorithm, were then used to make more accurate recommendations. SVD is now a cornerstone of most recommendation algorithms.
Brief history
2000 B.C.: Some of the oldest recorded algorithms in history are written on clay tablets in Babylonia.
820: Persian mathematician abu-Jaʽfar Muhammad ibn Mūsā al’Khwārizmī writes the Compendious Book on Calculation by Completion and Balancing, which later went on to introduce algebra to Europe. The word algorithm is derived from his name.
1842: English mathematician Ada Lovelace writes an algorithm for finding Bernoulli numbers using a hypothetical machine. Her algorithm is widely-considered the first computer algorithm.
1998: Sergey Brin and Lawrence Page introduce their notorious “PageRank” algorithm in an academic paper while they were students at Stanford University. They designed the algorithm as a way to bring “order to the web” by ranking search engine results (and then, of course, went on to start Google).
2016: The video-based social media network TikTok launches globally.
The problem with automating culture
In the almost 20 years since the Netflix Prize competition, recommendation algorithms have only gotten more sophisticated and so has the data they use.
TikTok — arguably the most powerful recommendation algorithm that is having a moment of reckoning in the U.S. — collects and uses data on the actual watch time of millions of videos from 1.5 billion active monthly users. The app has said publicly that its algorithm takes into account likes, comments, shares, and video watch time; however, investigations from journalists and experts have found that watch time is the key to the app’s eerily psychic algorithm. Every second you spend on the app gives the algorithm more data on the type of content that is most likely to keep you on the app — you don’t even have to hit the like button anymore.
And while these algorithms can be helpful, they are having unintended real-world implications. Youtube’s algorithm has improved, but it used to promote conspiracy videos. Facebook’s algorithm once prioritized angry emoji reactions, resulting in the spread of misinformation.
TikTok itself has been found to suppress content made by LGBTQ+ and disabled users. Researchers have also found that topics censored in China are underrepresented on TikTok compared to other social media platforms.
Listen up
Bought something you don’t need from an ad? Blame it on the algorithm. Disappointing singles on your dating app? Blame it on the algorithm. But are we right to always blame it?
In the season finale of the Quartz Obsession podcast — The algorithm: Letters of recommendation — Quartz reporter Bruce Gil tells host Gabriela Riccardi about the origins of the algorithms shaping what we do online.
🎧 Listen now on Spotify | Apple | Pandora
👓 Or, read the transcript
Quotable
“I think the problem with being surrounded by algorithmic recommendations is that it prevents us from being challenged and surprised a lot of the time, like everything is molded to our preferences that we’ve already expressed. The Spotify recommendations follow all the bands and genres that they know you like, that you engage with. We’re herded and shepherded toward experiences that we’re going to find comfortable enough.” — New Yorker writer Kyle Chayka on The Ezra Klein Show
Pop quiz
Which country has the most TikTok users?
A. United States
B. China
C. Indonesia
D. Brazil
Keep scrolling — the answer is at the bottom of this email.
By the digits
600 million: People subscribed to the music streaming service Spotify, which has been increasingly using AI to personalize music recommendations.
70%: Videos watched on YouTube that are driven by the platform’s recommendation algorithm.
100 million: Movie ratings Netflix made publicly available when it launched its Netflix Prize competition in 2006.
80%: TV shows and films watched on Netflix due to algorithmic recommendations.
40%: TikTok users in the U.S. who say that their “For you page” is extremely or very interesting to them.
Poll
Which app has the most time-sucking algorithm for you?
- TikTok — I can’t tear myself away from my feed
- YouTube — I can lose hours in a video rabbit hole
- Netflix — I can binge watch all day
Tell us about your experience with TikTok and other recommendation algorithms.
💬 Let’s talk!
In last week’s poll about video game remakes, 45% of you really just want something new instead of the same thing revised! But 21% of you are into reboots, 18% like remasters, and 16% actually do like a good remake.
🤔 What did you think of today’s email?
💡 What should we obsess over next?
Today’s email was written by Bruce Gil (who has found love through the algorithm) and edited by Morgan Haefner (is curious what algorithm her cheese purchase history would make).
The answer to the quiz is A., the United States. The U.S. has nearly 150 million TikTok users, the most of any country in the world. Indonesia and Brazil have the second and third largest TikTok audiences. TikTok is not available in China, and soon could face a ban in the U.S.
