OONI Highlights: 2017-2019

This post was originally published on the OTF website.

Back in 2012, the Open Observatory of Network Interference (OONI) was born with OTF support to create the first open methodologies that would enable anyone around the world to measure internet censorship and other forms of network interference.

Today, OONI comprises a global community that runs its censorship measurement software (OONI Probe) in more than 200 countries every month. Millions of measurements have been collected from tens of thousands of networks in 236 countries since 2012, all of which are openly published on OONI Explorer – arguably the largest publicly available resource on internet censorship to date.

Most of this growth is thanks to OTF support over the years.

In this post, we share highlights from what the OONI team accomplished between 2017 and 2019 with OTF Internet Freedom Fund support.

• Improved censorship measurement tools

• Expanded global coverage of censorship events

• Timely evidence of global censorship events

• Rapid response to emergent censorship events

• Global OONI censorship measurement community

Improved censorship measurement tools

Image: Comparison of OONI Probe mobile 1.0 with OONI Probe mobile 2.0 (both released between 2017-2019)

To enable the internet freedom community to measure censorship events around the world, we first needed to:

• Make OONI Probe available on more platforms (such as Android and Windows);

• Improve OONI Probe usability;

• Make OONI Probe more useful to the internet freedom community (by including new features based on community requests);

• Improve and expand upon OONI’s methodologies to collect richer measurements and detect more forms of internet censorship.

To this end, we released:

• New OONI Probe mobile app for Android and iOS. For the first time, it became possible to run OONI censorship measurement tests with the tap of a button!

• Revamped version of the OONI Probe mobile app. OONI Probe mobile 2.0 includes many new features, improved design and UX.

• New OONI Probe desktop app for Windows and macOS. This desktop app has a similar design and UX as the revamped OONI Probe mobile app.

• New OONI Probe Command Line Interface (CLI). This is for Linux users who may want to run OONI Probe on servers and contribute more stable measurements.

• New OONI Probe measurement engine. Previously, the OONI Probe mobile app was powered by the Measurement Kit library. To enable the integration of tests and iterate more quickly with higher confidence in code quality, we re-wrote the OONI Probe measurement engine from C++ to golang and released the OONI Probe engine.

• New OONI Probe tests. These include a new Telegram test (and rewrite in Go), OONI implementations of M-Lab’s NDT and DASH performance tests, ndt7 (improved speed test), a DASH rewrite using HTTPS, as well as follow-up measurements for DNS over TLS fallback, DNS over HTTPS fallback, and SNI blocking detection.

• Jafar: a censorship simulation tool. To ensure that our methods work as expected and to make OONI Probe more resilient, we built a censorship simulation tool to run experiments and improve the robustness of our tools.

• Methodology for investigating internet blackouts. We published a document which outlines our proposed methodology for investigating internet blackouts from the edge of the network. We also published a blog post which explains how to examine internet blackouts through existing public data sources.

Expanded global coverage of censorship events

Image: Monthly OONI measurement coverage worldwide

Every month, millions of OONI measurements are collected from thousands of networks in more than 200 countries, shedding light on censorship events around the world.

This expanded coverage is most likely due to the improvement of OONI’s censorship measurement tools. The following OONI measurement stats show a large uptick in February 2017, following the initial launch of the OONI Probe mobile app (which made censorship measurement much easier).

Image: OONI monthly measurement statistics (previously hosted on the OONI API)

Expanded measurement coverage means that more OONI Probe tests are run in more countries and on more networks, enabling the internet freedom community to detect many more cases of internet censorship around the world.

Between 2017 to 2019, we collaborated with our local partners (and other community members) on the publication of the following 32 research reports covering censorship events around the world:

• The State of Internet Censorship in Thailand

• The State of Internet Censorship in Myanmar

• The State of Internet Censorship in Indonesia

• #EgyptCensors: Evidence of recent censorship events in Egypt

• Measuring Internet Censorship in Cuba’s ParkNets

• Internet Censorship in Iran: Network Measurement Findings from 2014-2017

• Evidence of Internet Censorship during Catalonia’s Independence Referendum

• Internet Censorship in Pakistan: Findings from 2014-2017

• How Pakistan blocked news outlets, social media sites, and IM apps amidst protests

• Iran Protests: OONI data confirms censorship events (Part 1)

• Iran Protests: DPI blocking of Instagram (Part 2)

• Sierra Leone: Network disruptions amid 2018 runoff elections

• Nigeria: Measuring Internet Censorship

• Ethiopia: Verifying the unblocking of websites

• The State of Internet Censorship in Egypt

• Mali: Social media disruptions amid 2018 presidential election?

• South Sudan: Measuring Internet Censorship in the World’s Youngest Nation

• The State of Internet Censorship in Venezuela

• Uganda’s Social Media Tax through the lens of network measurements

• Internet disruption in Gabon amid coup attempt

• Zimbabwe protests: Social media blocking and internet blackouts

• From the blocking of Wikipedia to Social Media: Venezuela’s Political Crisis

• Cuba blocks independent media amid 2019 constitutional referendum

• Benin: Social media blocking and Internet blackout amid 2019 elections

• China is now blocking all language editions of Wikipedia

• Nigeria’s 2019 elections through the lens of network measurements

• Jordan: Measuring Facebook live-streaming interference during protests

• Ethiopia: From internet blackouts to the blocking of WhatsApp and Telegram

• Resurgence of Internet Censorship in Ethiopia: Blocking of WhatsApp, Facebook, and African Arguments

• Egypt blocks BBC and Alhurra: Expanding media censorship amid political unrest

• On the blocking of abortion rights websites: Women on Waves & Women on Web

• Iran’s nation-wide Internet blackout: Measurement data and technical observations

Timely evidence of global censorship events

Image: OONI Explorer (revamped in 2019)

Now, OONI measurements from around the world are openly published in near real-time. This means that as soon as anyone runs OONI Probe anywhere around the world, their test results are (automatically) openly published on OONI Explorer within minutes (unless if they have opted-out from publication). It is therefore now possible to track internet censorship events around the world in near real-time, thanks to the creation of a new fast-path pipeline.

Data analysts and researchers may be interested in the OONI API (launched in 2017) for downloading the raw measurements in JSON format to perform their own analysis.

Image: OONI API

Given that other tools (such as OONI Explorer) depend on the OONI API, the use of the API should be limited to light-weight queries (to avoid affecting the performance of tools that rely on it).

To support batch consumption of OONI data, we have made a full copy of the OONI PostgreSQL MetaDB available. This enables any third party to setup a read-only replica of the OONI MetaDB, allowing them to run heavy database queries on their own infrastructure for batch analysis. This is the same database that also powers the OONI API and OONI Explorer. To enable adoption, we published documentation that explains how to set-up a copy of the OONI MetaDB.

Thanks to the Amazon Open Data program, the whole OONI dataset can be fetched from the ooni-data Amazon S3 bucket, and we published a blog post explaining how to do this.

Rapid response to emergent censorship events

Image: OONI Run

To promote rapid response to emergent censorship events, we created a new platform, called OONI Run, to enable the coordination of customized OONI Probe testing.

Through OONI Run, you can generate a link based on specific websites that you are interested in testing, and share that link with OONI Probe users so that they can test those websites. This has been useful in various cases for measuring emergent censorship events around the world.

Last year, for example, IPYS Venezuela and Venezuela Inteligente coordinated an OONI Probe measurement campaign amid protests and political turbulence, sharing OONI Run links with Venezuelans to test YouTube, Instagram, and Twitter across the country. This enabled the rapid collection of relevant measurements, supporting the publication of a joint report providing evidence of social media blocking in Venezuela. Quite similarly, OONI Run links were used to coordinate testing in Egypt to collect evidence on the blocking of bbc.com and alhurra.com amid protests.

As social media, news websites, and circumvention tools often get blocked during political events around the world, we embedded several OONI Run buttons on the OONI website to enable the testing of such websites.

Image: OONI Run buttons embedded on OONI website

These sorts of buttons can be generated through the OONI Run platform.

Global OONI censorship measurement community

All of the above requires a community: humans who are interested in defending a free and open internet, and willing to run censorship measurement tests from local vantage points (despite potential risks).

To empower censorship research participation in the internet freedom community, we established 17 new partnerships (between 2017 to 2019) with the following organizations:

• Open Culture Foundation (Taiwan)

• Software Freedom Law Centre (India)

• Association for Freedom of Thought and Expression (Egypt)

• Jordan Open Source Association (Jordan)

• Paradigm Initiative (Nigeria)

• Digital Society of Zimbabwe (Zimbabwe)

• Tuwindi Foundation (Mali)

• ASUTIC (Senegal)

• COMPSUDEV (Cameroon)

• The Advocates for Human Rights and Democracy (South Sudan)

• Campaign for Human Rights & Development International (Sierra Leone)

• Instituto Prensa y Sociedad (IPYS) Venezuela (Venezuela)

• Asociación por los Derechos Civiles (Argentina)

• Fundación Karisma (Colombia)

• Red en Defensa de los Derechos Digitales (Mexico)

• TEDIC (Paraguay)

• Fundación Internet Bolivia (Bolivia)

We started establishing partnerships with digital rights organizations in mid-2016, and the new partnerships listed above expand our network to 27 partnerships overall. Through these partnerships, we aim to support local efforts in increasing transparency of internet censorship. To this end, we collaborate on OONI network measurement and provide OONI data analysis support.

Between 2017 to 2019, we collaborated with our partners (and other community members) on updating 20 Citizen Lab test lists for the following countries: Senegal, Bolivia, Mexico, Paraguay, Tanzania, Algeria, Vietnam, Indonesia, Cameroon, Democratic Republic of Congo (DRC), Colombia, Argentina, Jordan, Nigeria, South Sudan, Venezuela, Zimbabwe, Sierra Leone, Egypt, and Mali.

To support local OONI community engagement efforts, we created and published several resources, such as OONI workshop slides, a Frequently Asked Questions (FAQ) section, and an OONI Glossary. We also started an OONI Community Interview series and created a short documentary on internet censorship in Cuba.

Thanks to OTF Community Lab support, we had the opportunity to host the OONI Partner Gathering in 2017: a two-day event that brought all of our partners together to share skills and knowledge on censorship measurement research. We subsequently held several other community gatherings, such as our monthly online meetings and in-person meetups at international conferences and events (such as RightsCon and the Internet Freedom Festival).

The OONI team is immensely grateful to the OTF for supporting the growth of the OONI-verse. We look forward to continuing to support the internet freedom community in the fight against internet censorship.