Archive for the 'english' Category

postwhite

I recently updated my small mailserver and finally configured DKIM. But another change was easier and still had more impact: installing postwhite. This little tool takes a list of mail domains, then uses their SPF records to derive a list of their outgoing mail servers, then writes this list into a postscreen whitelist configuration. The current default setting contains 43 domains and generates a whitelist with nearly 2000 lines (each containing an IP or subnet). Everything is nicely scripted and can run as a nightly cronjob.

This setup eliminates my biggest problem with greylisting, which is Office356. Their combination of long email resubmit intervals and using multiple cluster servers for delivery attemps always lead to long delays before I received email from Microsoft or any company using Office356. (BTW, I really like greylisting but this is its biggest design problem: it works for single SMTP servers and enforces certain behaviour, but does not and can not consider clusters.)

Links 2018-11-12

More about microservices and Docker …

Links 2018-10-29

Some computing history …

Python’s concurrent.futures module

For some reason I wanted to improve performance of a small data driven Python program and tried to parallelize it. These are a few learnings to keep around for next time.

I nearly started with the very basics, defining my own threads as well as task and result queues. But then I found the very useful concurrent.futures module which provides a high-level interface to distribute tasks to both threads and processes.

I still made the mistake to start with threads. Everything worked nicely and the tool ran along with four worker threads — but every thread received 25% of CPU time and the overall runtime did not improve. I realized I had forgotten about Python’s Global Interpreter Lock (GIL).
The GIL basically prevents performance improvement using multithreading (at least of CPU-bound tasks, it is still useful for I/O). More information about the GIL:

So I had to switch to multiprocessing instead. The switch itself is really easy because it is nearly completely hidden inside concurrent.futures, I only had to replace the initialization of the ThreadPoolExecutor() with a ProcessPoolExecutor().

But with multiple processes I can no longer share variable values. Everything, including the called function itself, has to be pickled and send to the subprocess.
This required some refactoring, as I had to move the function to the module top-level (as local functions cannot be pickled) and then tried to find a good minimal set of parameters and return values in order to reduce the data transfer between the processes.

I saved my code examples for different concurrent.futures invocations as a gist for later reference: mschuett/concurrent.py

Along the way I also tried the asyncio module for “Asynchronous I/O, event loop, coroutines and tasks”. That one is also quite interesting, but as the name suggests it is focussed on I/O and coroutines in a single thread; functions you need for a network server. For my use case it is not useful, because asyncio does not help to utilize a second CPU core.

Links 2018-10-22

Links 2017-07-16

Even more on programming and architecture.

Links 2017-05-21

  • The Dark Secret at the Heart of AI
    No one really knows how the most advanced algorithms do what they do. That could be a problem.
  • The Threat
    There were only a few application areas [of information security] that people really worried about thirty years ago: diplomatic and military communications at one end, and the security of things like cash machines at the other. As we’ve gone about putting computers and communications into just about everything that you can buy for more than ten bucks that you don’t eat or drink, the field has grown.
  • Inside Chrome: The Secret Project to Crush IE and Remake the Web
    Why is Google building a browser? A better question is, why did it take so long for Google to build a browser?
  • WWW: The Way We Were
    The final two episodes of Halt and Catch Fire aired last night. […] The Web that they are talking about on the show, the open Web, is ailing, dying. It was like listening to a eulogy at a funeral, this thing that I love, poured the best of my self into, gone forever.
  • Everything Is Fucked and I’m Pretty Sure It’s the Internet’s Fault
    Democracy relies on trust. Rule of law requires trust. If we lose our trust in our institutions, then those institutions will either crumble or turn cancerous. But the internet lines up incentives in such a way that it makes it profitable to breed distrust.
  • The Real Name Fallacy
    The idea that anonymity is the real problem with the internet is based in part on misreadings of theories formed more than thirty years ago.

Links 2017-05-17

  • AWS Security Primer
    I was preparing some AWS Security related training. Soon, I realized that this topic is too huge to fit into my brain. So I structured my thoughts in a mind map. Within a couple of minutes1 I came up with this.
  • Takeaways From the ServerlessConf 2017
    Serverless computing basically adds another layer of abstraction on top of cloud infrastructure so that the developer doesn’t need to worry about servers, including virtual ones.
  • How Much Does It Cost To Run A Serverless API on AWS? · Alestic.com
    Folks tend to be curious about how much real projects cost to run on AWS, so here’s a real example with breakdowns by AWS service and feature. During this month, TimerCheck.io service processed over 2 million API requests.
  • How Google Is Challenging AWS
    Big companies are often criticized for having “missed” the future — from the comfortable perch of a present where said future has come to pass, of course — but while the future is still the future incumbents are first more often than not.
  • Best Practices for Building a Microservice Architecture
    A microservice architecture shifts around complexity. Instead of a single complex system, you have a bunch of simple services with complex interactions. Our goal is to keep the complexity in check.
  • How the Internet works: Submarine fibre, brains in jars, and coaxial cables
    Have you ever thought about how that cat picture actually gets from a server in Oregon to your PC in London? We’re not simply talking about the wonders of TCP/IP or pervasive Wi-Fi hotspots, though those are vitally important as well. No, we’re talking about the big infrastructure: the huge submarine cables, the vast landing sites and data centres with their massively redundant power systems, and the elephantine, labyrinthine last-mile networks that actually hook billions of us to the Internet.