THIS TOO SHALL PASS

Written by Francois Fernando, a software craftsman, tinkerer and lifelong learner.

Exposing MQTT (or other services) on MicroK8S on a home network

MicroK8S is the most user friendly kubernetes distribution I have found for home use. I have been looking for a simple guide on how to…

Why Confluent Kafka Avro encoding bothers me!

Recently I have been working to get PySpark to read from a Kafka topic. We use a Confluent based deployment of Kafka with Schema Registry…

Workaround for Spark Kubernetes client version incompatibility

When we upgraded the Kubernetes cluster from 1.11 to 1.16 we immediately ran into an issue with our PySpark (version 2.4.1) based Jobs. They…

Running PySpark on Kubernetes

Documenting the steps I had to go through getting PySpark running on an on premise Kubernetes (K8S) cluster on OpenStack. This cluster has…

From Wordpress to a static site in a weekend

I have been interested in static site generators like Jekyll for a while. But until now I haven’t had time to try one. While migrating a…

3D printing rescues the dishwasher

Our seven year old dishwasher has shown to start it’s age. Three of the lower rack wheels started to come off from rack due to breaking off…

Trigger DAG run from a task in another DAG

When writing airflow jobs sometimes it’s necessary to trigger a DAG run from a task in another DAG. Airfow provides TriggerDagRunOperator…

Embedding Apache Spark in an Application

Recently I wanted to do some data encryption on a data set while preserving the format. After looking at few things I settled on this…

Databases are global, shared, mutable state

Databases are global, shared, mutable state. That’s the way it has been since the 1960s, and no amount of NoSQL has changed that. However…

Does data locality matter any more?

Yesterday went to the Spark Meetup in Sydney. One of the speakers Mike Seddon mentioned an important point. Most distributed processing…