Highlights: 10 Years of OONI

Maria Xynou 2022-12-05

Today is OONI’s 10th anniversary!

As of today, 10 years ago, the first OONI measurement was published. Today, 10 years later, more than a billion OONI measurements have been published, shedding light on internet censorship worldwide.

In this post, we share some OONI highlights from the past 10 years, as well as some of our future plans.

You can start off by getting a glimpse of “OONI in 10 years” through the following animation, which shares some highlights from the past decade.

Animation: Created by Robotina.

About OONI

More than ten years ago, some developers at the Tor Project were investigating how internet censorship was being implemented on certain networks. Their main motivation was to understand the blocking of Tor, but they were also curious to know more broadly how censorship was affecting other services as well.

While carrying out these investigations, they developed a set of scripts that they would ask tech-savvy people in regions affected by censorship to run. OONI was born out of the desire and need to further systematize this collection of one-off network measurements so that anyone, without coordination from us, could run them independently and the results would be published for the world to see.

With this in mind, we worked on the following over the last decade:

In the following sections, we share key OONI highlights from the past decade, as well as some of our plans for the future.

Highlights

Free software tools for measuring internet censorship

Network measurement tests

Over the past decade, we have created numerous network measurement tests designed to measure various forms of internet censorship.

These include tests designed to measure:

The above experiments have been integrated into our OONI Probe app, enabling users to run these tests on networks worldwide. We also integrated the NDT and DASH performance tests, developed by, and in collaboration with, the Nexa Center for Internet and Society and Measurement Lab (M-Lab). To encourage community members to contribute their own experiments to OONI Probe, we published a tutorial explaining how to write OONI Probe tests.

More recently, we have been researching new methods to improve how we measure website blocking. Our new methods enumerate all forms of website blocking, include support for HTTP/3 and a follow-up SNI filtering test, enabling the collection of richer data. We aim to incrementally ship these new methods as part of our Web Connectivity experiment over the next year.

OONI Probe

You can measure various forms of internet censorship with OONI Probe, currently available for Android, iOS, F-Droid, Linux, Windows and macOS.

OONI Probe started off as a python command line tool for Linux and macOS, with OONI Probe 1.0.0 being released in February 2014. Initially, OONI Probe was mainly run by technologists, since it required familiarity with the command line. To enable researchers and human rights defenders to run OONI Probe, we released OONI Probe 2.0.0 with a web-based user interface.

During the same year (2016), we also established the OONI Partnership Program to collaborate with digital rights organizations on measuring internet censorship around the world. To enable our partners to contribute stable measurements, we released an OONI Probe distribution for Raspberry Pis (“Lepidopter”).

Image: OONI Probe for Raspberry Pi (credit: Sinar Project).

While “Lepidopter” played an important role in boosting OONI measurement coverage (with daily measurements collected automatically), shipping (and troubleshooting) Raspberry Pis was not sustainable for our small team in the long-run. We therefore aimed to instead prioritize our efforts on building OONI Probe apps that anyone can easily install and run, without requiring any technical knowledge.

Notably, we launched the OONI Probe mobile app (for Android, F-Droid and iOS) in February 2017, bringing censorship measurement to hundreds of thousands of people worldwide. This launch was a major milestone because, for the first time, anyone could independently measure internet censorship – without requiring any technical knowledge or expertise. For the first time, anyone with a smartphone (most internet users worldwide) could easily measure and contribute data on internet censorship with the tap of a button! And based on community feedback, we launched a revamped version of OONI Probe Mobile (with a major UI overhaul) two years later (2019).

Image: Comparison of OONI Probe Mobile 1.0 (old) with OONI Probe Mobile 2.0 (new).

We subsequently improved the UI of OONI Probe Mobile further.

While OONI Probe Mobile played a significant role in boosting OONI measurement coverage worldwide, desktop users still had to use the terminal to run OONI Probe. To enable people to run OONI Probe on as many platforms as possible, we launched an OONI Probe desktop app for Windows and macOS in April 2020.

With the OONI Probe mobile and desktop apps, anyone can measure internet censorship on multiple platforms, with the click of a button! To enable technologists to install OONI Probe on servers and contribute more regular measurements, we also released a new OONI Probe Command Line Interface (CLI) for Linux and macOS.

In response to community requests, we built a browser-based version of OONI Probe (“OONI Probe Web”) which we plan to launch in 2023.

Overall, to improve the monitoring of website censorship around the world, we created a “smart URL list system” which ensures that more relevant and recently merged URLs are prioritized for testing. This has supported rapid response efforts worldwide, ensuring that newly blocked URLs are immediately tested.

OONI Run

Running OONI Probe on your network is not enough. Often, we hear of censorship in different countries (where we’re not physically based), and there’s the need to coordinate with OONI Probe users in other countries for censorship testing.

To support community efforts aimed at coordinating censorship testing, we launched OONI Run in September 2017. OONI Run is a platform that you can use to generate mobile deep links and widget code to coordinate OONI Probe testing. Initially, OONI Run supported multiple OONI Probe tests, but based on community feedback, we limited the platform to website testing and made other improvements.

Over the past 5 years, community members have used OONI Run to coordinate censorship testing with their local communities in countries like Venezuela and Ukraine. OONI Run has also been used to coordinate the testing of specific websites that risk being blocked during political events, such as elections. Sinar Project, for example, recently coordinated an OONI Probe testing campaign (using OONI Run links and widget code) in Malaysia during the country’s 2022 elections.

Real-time open data on internet censorship

As of 1st October 2020, OONI Probe measurements from around the world are published in real-time! This means that as soon as you run OONI Probe (anywhere in the world), your test results are immediately published as open data within seconds.

This is an important milestone because it means that the public can track (and rapidly respond to) censorship events around the world as they emerge. This is also the first time that any project publishes open data on internet censorship in real-time.

Migrating our infrastructure to ClickHouse enabled the real-time publication of OONI measurements, supporting human rights defenders to rapidly monitor and respond to internet censorship around the world.

OONI Explorer

To enable researchers, journalists, and human rights defenders to explore OONI measurements and investigate internet censorship, we launched OONI Explorer in March 2016. Long-term OONI community members probably remember the original version of OONI Explorer (which included a global map):

Image: First version of OONI Explorer.

To better address community feedback and needs, we launched a revamped version of OONI Explorer in September 2019!

The new version of OONI Explorer includes a powerful Search Tool which enables you to easily check which websites are automatically confirmed blocked around the world.

Today, OONI Explorer hosts more than 1 billion measurements collected from 25 thousand networks in 241 countries and territories. It is the largest open dataset on internet censorship (based on measurements collected from local vantage points) to date.

Measurement Aggregation Toolkit (MAT)

To identify censorship trends and patterns, it is necessary to look at many measurements at once (in aggregate). To this end, we launched the OONI Measurement Aggregation Toolkit (MAT) in April 2022, which enables you to track internet censorship worldwide in real-time and create charts based on aggregate views of OONI data.

Through the MAT, you can plot charts to compare censorship between countries and networks, you can check the blocking of websites and apps globally or in specific countries, and you can check which categories of websites (e.g. news media, LGBTIQ, environmental, human rights) are blocked in each country.

For example, the MAT shows that Iran started blocking Instagram on 21st September 2022, during the ongoing protests.

OONI API

To enable researchers and data scientists to analyze OONI data, we launched the OONI API in September 2017.

You can download raw OONI data in JSON format through the OONI API and perform your own analysis. We recommend using the OONI API for light-weight queries. For batch analysis, we recommend fetching OONI data from the Amazon S3 bucket.

Research on internet censorship

Over the past decade, we published 75 reports documenting internet censorship worldwide. Almost all of these reports involve the analysis of OONI data, while many were produced in collaboration with our partners.

These reports include research on the blocking of LGBTIQ websites and reproductive rights websites, on the blocking of the Signal Private Messenger App around the world, and on increased censorship events that emerged in Myanmar following the 2021 military coup and in Russia following the war in Ukraine. More recently, we published a report documenting new censorship events that emerged in Iran amid the ongoing protests. We also published a technical multi-stakeholder report combining multiple measurement datasets in investigating internet censorship events in Iran. As these reports present and contextualize OONI data findings, they have supported a wide range of research and advocacy efforts over the years.

Beyond research reports published on our website, we also co-authored several research papers:

Resources for community participation in censorship measurement research

Online OONI training courses

To support OONI community engagement efforts worldwide, we created several OONI training courses in collaboration with our partners.

The most comprehensive OONI training course was recently launched on Small Media’s Advocacy Assembly platform.

Image: OONI training course on Advocacy Assembly.

This 90-minute free, online training course provides a deep-dive into measuring internet censorship with OONI Probe, and using OONI Explorer to access and interpret real-time OONI data collected from around the world. Designed for human rights defenders, activists, journalists, and researchers, the course includes a mix of videos, screencasts, slides, quizzes and hands-on exercises. It also features case study videos from OONI community members, and the course is available in English, Arabic, Spanish, and Farsi.

We previously collaborated with Internews on creating and facilitating OONI training modules for the OPTIMA Network Measurement Training Program for Advocates. We also collaborated with Asuntos del Sur on creating a micro-course on internet censorship (including a video about OONI Probe) for their Plataforma Inteligencia de Riesgos.

Test Lists Editor

Since 2014, we have collaborated with the Citizen Lab on updating the lists of websites (“test lists”) measured by OONI Probe and other censorship measurement projects. In recent years, we also collaborated closely with Netalitica, who have led research efforts in updating the test lists.

To enable community members to contribute websites for censorship testing (without having to use GitHub), we launched a Test Lists Editor.

Image: Test Lists Editor.

Through this web platform, community members can add websites for OONI Probe censorship testing, edit existing entries (for example, to change a website’s categorization or to update a URL), or propose the deletion of URLs. By enabling community members to dynamically contribute to and update the test lists, we aim to improve the quality of website censorship testing, supporting rapid response efforts to emergent censorship events.

User guides and documentation

To enable global community use of OONI tools and data, we published documentation and several user guides over the years.

User guides

Documentation

Global OONI censorship measurement community

OONI Partners

Since the establishment of the OONI Partnership Program in 2016, we have had the opportunity to form partnerships with 41 digital rights organizations. Many of our partners include local digital rights organizations in Africa, Latin America, Asia, and the Middle East. These partners serve as OONI ambassadors in their countries/regions, leading OONI community engagement efforts and using OONI tools and data as part of their work.

Notably, some of our partners run entire projects dedicated to measuring Internet censorship through the use of OONI tools and OONI data. Sinar Project, for example, runs the Internet Monitoring Action Project (iMAP) which investigates Internet censorship in Malaysia, Indonesia, Philippines, Cambodia, Vietnam, Thailand, Hong Kong, Myanmar and India through the use of OONI tools and OONI data. Venezuela Inteligente have used OONI tools and data as part of their Internet censorship reporting over the last 8 years.

In July 2017, we organized and hosted the OONI Partner Gathering, which brought all of our partners together in Toronto for a 2-event on OONI censorship measurement. In June 2020, we hosted the online Internet Measurement Village, which featured 18 live-streamed presentations by our partners and other community members. In July 2021, we hosted two 3-day OONI Partner Training events, both of which were hosted entirely online (due to the COVID-19 pandemic).

KeepItOn campaign

Since 2016, OONI has been a member of the #KeepItOn campaign: a global coalition of human rights organizations fighting internet shutdowns around the world. We have since worked closely with the campaign by supporting the coordination of censorship measurement efforts (particularly leading up to and during political events, which often trigger new censorship events) and by providing relevant OONI data in support of advocacy efforts around the world.

Localization of OONI tools

Thanks to the Localization Lab community, OONI Probe has been translated to numerous languages, including Farsi, Arabic, Chinese, Russian, Hindi, Thai, Indonesian, Swahili, Spanish, and French – among many other languages. Similarly, OONI Explorer is being translated to numerous languages, and the community recently started translating the OONI Probe Mobile and OONI Probe Desktop user guides as well.

We thank the Localization Lab for bringing OONI censorship measurement to communities worldwide!

Community use of OONI data

Over the past decade, OONI data has supported a wide range of research and advocacy efforts.

OONI data has supported research on internet censorship in Iran, Egypt, Malaysia, the Philippines, India, Venezuela, Rwanda, Uganda, Lebanon, Myanmar, Azerbaijan, Ukraine, Russia and Crimea (among many other countries). Freedom House has cited OONI data in many of their annual Freedom on the Net country reports. OONI data has also supported academic papers, such as research on China’s DNS censorship, global CDN geoblocking, global I2P censorship, and on the deployment of network censorship filters at a global scale.

Internet Society (ISOC) includes OONI data in their Pulse Internet Shutdowns project, which provides a timeline of blocking events and internet shutdowns around the world. Harvard’s Berkman Klein Center integrated OONI data into their AccessCheck project. iThena (a distributed computation and measurement project based on the Berkeley Open Infrastructure for Network Computing (BOINC) platform) integrated OONI Probe into their platform, boosting OONI measurement coverage worldwide.

Advocacy efforts around the world have made use of OONI data. Access Now, for example, have cited OONI data as part of advocacy challenging censorship events in Tanzania, while Internet Sans Frontieres have cited OONI data as part of advocacy challenging social media censorship in Mali. The global #KeepItOn campaign recommends the use of OONI tools and data as part of their advocacy toolkits, and they regularly cite OONI data as part of their annual advocacy reports. A few years ago, OONI data on the blocking of websites amid protests in Pakistan supported the petition of local organizations who challenged the legality of these blocks at the Islamabad High Court.

Journalists around the world have also made use of OONI data when reporting on emergent censorship events. For example, OONI data is cited in news articles by Wired, BBC, CNN, CBC News, CNET, The Intercept, Wall Street Journal, Deutsche Welle, Taggesspiegel, Mada Masr, Al Araby, Time, and Africa Times (among many others). The Committee to Protect Journalists (CPJ) also published an article which discusses why journalists should use OONI data.

Future goals

Ultimately, our goal is to ensure that internet censorship is well-documented and rapidly addressed so that human rights are protected.

Over the past decade, community feedback has shaped the specific activities that we have carried out towards this goal, as we aim to ensure that the tools we build meet community needs. We thank all community members who shared feedback with us over the years, and we thank community members who recently shared more feedback through our survey.

Based on long-term community feedback and strategic planning, our future goals (for at least the next 2 years) include:

Below we list some of the main activities through which we aim to achieve each of the above goals.

Expanding OONI’s censorship detection capabilities

Enabling the discovery of censorship findings through real-time OONI data

Empowering global community participation in censorship measurement research and advocacy

Acknowledgements

We thank the Tor Project for supporting OONI in its infancy, and helping our project grow. We also thank all of our funders for supporting OONI’s work over the past decade.

From the bottom of our hearts, we thank all OONI Probe users out there for contributing measurements – especially those in high-risk environments.

We thank all community members who have been part of OONI’s journey over the past 10 years, contributing to the growth of the project in so many different ways – participating in usability studies, sharing feedback, opening GitHub tickets to report bugs and share feature requests, opening GitHub pull requests to contribute to or improve OONI’s code, updating test lists, participating in monthly community meetings, sharing resources and updates on community channels, localizing OONI tools, teaching OONI tools to their local communities, using OONI data as part of their research and advocacy… and so much more!

We thank each and every one of you.

We’re excited for the next 10 years!