Links 2018-11-12

More about microservices and Docker …

Links 2018-10-29

Some computing history …

Python’s concurrent.futures module

For some reason I wanted to improve performance of a small data driven Python program and tried to parallelize it. These are a few learnings to keep around for next time.

I nearly started with the very basics, defining my own threads as well as task and result queues. But then I found the very useful concurrent.futures module which provides a high-level interface to distribute tasks to both threads and processes.

I still made the mistake to start with threads. Everything worked nicely and the tool ran along with four worker threads — but every thread received 25% of CPU time and the overall runtime did not improve. I realized I had forgotten about Python’s Global Interpreter Lock (GIL).
The GIL basically prevents performance improvement using multithreading (at least of CPU-bound tasks, it is still useful for I/O). More information about the GIL:

So I had to switch to multiprocessing instead. The switch itself is really easy because it is nearly completely hidden inside concurrent.futures, I only had to replace the initialization of the ThreadPoolExecutor() with a ProcessPoolExecutor().

But with multiple processes I can no longer share variable values. Everything, including the called function itself, has to be pickled and send to the subprocess.
This required some refactoring, as I had to move the function to the module top-level (as local functions cannot be pickled) and then tried to find a good minimal set of parameters and return values in order to reduce the data transfer between the processes.

I saved my code examples for different concurrent.futures invocations as a gist for later reference: mschuett/

Along the way I also tried the asyncio module for “Asynchronous I/O, event loop, coroutines and tasks”. That one is also quite interesting, but as the name suggests it is focussed on I/O and coroutines in a single thread; functions you need for a network server. For my use case it is not useful, because asyncio does not help to utilize a second CPU core.

Links 2018-10-22

Links 2018-03-04

Links 2017-09-22

Politisches zum Wahl-Wochenende…

  • Politik als Luxus
    Wer sich nicht einmal eigene Kinder zutraut hat keine Nerven für den Luxus Politik.
  • Schulzzug: Veränderung! Ach nee, doch nicht.
    Allerdings ist die Wahl nur solange bizarr, wie man davon ausgeht, die Menschen wählten, um etwas zu bekommen, und seien überzeugt, CDU und FDP könnten dieses Etwas geben.
  • Perspektivlosigkeiten
    tl;dr: Die Wahl ist gelaufen, es gibt nichts zu wählen. Ich geh trotzdem wählen, weil ich keinen Trump-Effekt haben will.
  • Seit ich bei #G20 war, habe ich Angst
    Ich habe mich noch nie so unwohl im Nachgang meiner Arbeit gefühlt, wie jetzt nach den G20-Protesten. Und das liegt nicht vordergründig an den Aktionen militanter Gewalttäter im Schanzenviertel.
  • Des Faschodrives neuste bürgerliche Fassade
    Man kann von einer weltweiten Lust am Totalitären sprechen, am Autokratischen, an uneingegrenzter staatlicher Gewalt und einfachem Freund/Feind-Denken.
  • Chronik des Überwachungsstaates
    Vor allem seit der Jahrtausendwende erlassen Bundesregierungen Gesetze, die kontinuierlich Grund- und Freiheitsrechte abbauen, indem Überwachung ausgedehnt wird.

Links 2017-07-16

Even more on programming and architecture.

Links 2017-05-21

  • The Dark Secret at the Heart of AI
    No one really knows how the most advanced algorithms do what they do. That could be a problem.
  • The Threat
    There were only a few application areas [of information security] that people really worried about thirty years ago: diplomatic and military communications at one end, and the security of things like cash machines at the other. As we’ve gone about putting computers and communications into just about everything that you can buy for more than ten bucks that you don’t eat or drink, the field has grown.
  • Inside Chrome: The Secret Project to Crush IE and Remake the Web
    Why is Google building a browser? A better question is, why did it take so long for Google to build a browser?
  • WWW: The Way We Were
    The final two episodes of Halt and Catch Fire aired last night. […] The Web that they are talking about on the show, the open Web, is ailing, dying. It was like listening to a eulogy at a funeral, this thing that I love, poured the best of my self into, gone forever.
  • Everything Is Fucked and I’m Pretty Sure It’s the Internet’s Fault
    Democracy relies on trust. Rule of law requires trust. If we lose our trust in our institutions, then those institutions will either crumble or turn cancerous. But the internet lines up incentives in such a way that it makes it profitable to breed distrust.
  • The Real Name Fallacy
    The idea that anonymity is the real problem with the internet is based in part on misreadings of theories formed more than thirty years ago.