Teaching Open Source Software in North Korea

posted on 2021-06-24

In 2016, I went to North Korea to teach a class of masters students how to contribute to open source software. Here’s an image from one of my lectures:

Mike Izbicki teaching open source software in North Korea

Mike Izbicki teaching open source software in North Korea

As part of the class, students were required to submit patches to a project of their choosing, and I want to share the stories of how two of these patches landed into the popular machine learning libraries mlpack and vowpal wabbit. I believe these examples highlight how academic collaboration between North Koreans and Americans can benefit ordinary citizens of both countries and improve diplomatic relations.

The mlpack patch

One of the students was working on a “vision-based vehicle detection system” for his masters thesis. In this problem, we are given a live feed from a video camera mounted near a road, and the goal is to count the number of cars and trucks that pass by. This is a fairly standard machine vision problem that students around the world regularly implement, and the output looks something like:

(This image unfortunately isn’t from the student’s project, but is instead taken from https://github.com/ahmetozlu/vehicle_counting_tensorflow.)

Apparently, the North Korean department of transportation had directed the student to work on this problem because traffic in Pyongyang was growing rapidly. Visitors to Pyongyang in the 1990s would often remark about the lack of cars on the roads, but these days the city is bustling with traffic. I guess Pyongyang’s famous Traffic Girls could use some automated help keeping traffic flowing.

Graduate students in North Korea have unfiltered internet access, and the student had used this access to download the mlpack machine learning library in order to implement their vehicle detection system. They chose to use mlpack because it was written in C++, and that was the main language they had been taught in their undergrad university courses. But they were encountering a very serious problem: their computer was old and did not have enough memory to compile the library!

After an hour or so of debugging with the student, we narrowed down the problem to mlpack’s use of C++ templates. The mlpack library uses templates extensively throughout its code in order to enable generic programming with no runtime overhead. This use of templates has earned mlpack a well-deserved reputation for running models very fast with low memory overhead, but the downside is that compilation is slow and memory intensive. At the time, compilation consumed a peak of about 2GB of RAM, but the North Korean student’s laptop only had 1GB of RAM available.

The student finally managed to get mlpack to compile by greatly optimizing the compilation procedure. The original code contained hundreds of unnecessary #include statements spread across the project, and the fix was simply to remove this dead code. You can view the actual commit on github. The fix sped up compilation by about 25% and more than halved memory consumption. The tens of thousands of people and companies who have used mlpack since then are all benefiting from this North Korean student’s excellent work.

The vowpal wabbit patch

Many of the masters students in my class had not yet selected a thesis topic, and so I encouraged one student to work on twitter sentiment analysis. In this problem, we are given a tweet like

https://twitter.com/senatormenendez/status/1006551464406323201?lang=ca

and we must classify the tweet as having either positive or negative sentiment towards a topic. For example, the tweet above has a negative sentiment towards the 2018 Singapore Summit between President Trump and Chairman Kim Jong-Un. By analyzing thousands (or millions/billions) of tweets this way, we can determine how different communities feel about a particular topic. Again, this is an entirely routine task implemented by students around the world. But for the North Korean students, this task was remarkable.

The students had all heard of twitter before, but they didn’t use it. Even though their computers had a direct, unfiltered connection to the internet, they were not allowed to create accounts on social media websites. The reason—as it was explained to me—is because the United States controls most internet infrastructure (including websites like Twitter), and through programs like the NSA’s PRISM and the Army’s Cyber Command is spying on and manipulating social media. The United States and North Korea never signed a peace treaty after the Korean War, and so North Koreans still very much worry about being attacked by the US. Not creating social media accounts was one of the “defenses” that these North Korean students were required to use in order to limit the effect of potential “American cyberattacks”. Fortunately, studying twitter is one of my research areas, and I had brought some reasonably sized data sets with me for the students to analyze.

What’s remarkable about this project—as one of my North Korean colleagues liked to remind me—is that it was the first time a North Korean student had ever analyzed twitter data. And analyzing twitter data would soon have geopolitical importance: Less than a year after this project was started, Donald Trump was elected president of the United States, and twitter became one of the primary tools that his administration would use to announce its foreign policy to the world. Fortunately, by the time Trump gave twitter this pseudo-official status, at least a small number of North Koreans had experience analyzing twitter data. They could use this experience to better understand both Trump’s tweets and the responses sent by millions of Americans. The North Korean government now recognizes the importance of using social media to understand American policy, and has recently created a new foreign ministry dedicated to analyzing US intentions through social media and other public information sources. And this is great news for both countries! The United States is built on a system of transparency because we actively want everyone—including North Koreans—to understand how our democracy functions and how to best negotiate with us to achieve shared goals.

This interaction between scientific exchanges and diplomacy is called science diplomacy, and it was instrumental in helping the US and Soviet Union negotiate successful limitations on nuclear weapons systems during the cold war. I believe my work teaching open source software in North Korea helps demonstrate that this science diplomacy model can also be successfully applied to US-North Korean negotiations as well.

Now back to the student’s open source contribution. I recommended that the student use vowpal wabbit to perform the analysis, since this is a great tool for analyzing large scale text datasets. The student successfully downloaded the code, compiled it, and analyzed the sentiment of a few thousand tweets. In this case, the code worked fine on the student’s computer without modification. But for the class on open source software, the student was still required to submit a patch.

He found an open issue on github asking for the ability to correct how the intercept term interacts with L2 regularization when training linear models, and submitted a patch to add this behavior. (The pull requests for both projects were submitted from my github account since github is considered social media, and the North Korean students weren’t allowed to create social media accounts.) A bug was later found in this patch, and a subsequent patch was added to fix the issue. This back-and-forth is exactly how open source software development is supposed to work, and I find it amazing that open source software lets ordinary people from around the world find common purpose building awesome software even across seemingly irreconcilable political differences.

The future of open source in North Korea

AFAIK, the patches submitted for this class were the first ever open source contributions to come from North Korea; unfortunately, they were also the last.

I had organized another trip to North Korea the following year (2017) that would have brought several other instructors to teach about open source software, but President Trump banned Americans from travelling to North Korea. So I and the other instructors could no longer meet with North Korean students, and there was no one to teach them how to contribute to open source or encourage them to do so.

President Biden has recently announced his policy of “practical diplomacy” with North Korea. But the details of this policy are not yet clear, and the travel ban remains in effect. As long as this policy remains in effect, I and other American instructors will not be able to help North Koreans contribute to open source. So Americans will not benefit from North Koreans fixing bugs in our code, and the science diplomacy that effectively reduced tensions between the US and Soviet Union cannot be used as a tool to reduce tensions between the US and North Korea.