Workaround for Spark Kubernetes client version incompatibility

January 08, 2020

When we upgraded the Kubernetes cluster from 1.11 to 1.16 we immediately ran into an issue with our PySpark (version 2.4.1) based Jobs. They started failing where spark context was being created. The container log contained following error messages.

20/01/07 05:34:19 INFO ExecutorPodsAllocator: Going to request 2 executors from Kubernetes. 
20/01/07 05:34:20 WARN WatchConnectionManager: Exec Failure: HTTP 403, Status: 403 - 
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden' 
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:216) 
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:183) 
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141) 
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
	at java.lang.Thread.run(Thread.java:748)
20/01/07 05:34:20 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.) 
20/01/07 05:34:20 ERROR SparkContext: Error initializing SparkContext. 
io.fabric8.kubernetes.client.KubernetesClientException: 
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onFailure(WatchConnectionManager.java:201) 
	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:543) 
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:185) 
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141) 
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
	at java.lang.Thread.run(Thread.java:748)

When looking at the above stack trace my suspicion was that due to kubernetes cluster upgrade, an earlier API endpoint may be no longer available. You can also see from above stack trace, Spark uses fabric8 kubernetes client under the hood.

I wanted to establish the version of fabric8 K8S client version used in Spark 2.4.1 and this could be done by listing the contents of /opt/spark/jars/ dir in docker container. I found following 3 jar files bundled with Spark.

kubernetes-client-4.1.2.jar
kubernetes-model-4.1.2.jar
kubernetes-model-common-4.1.2.jar

It was a blind guess that if I replace the jar files in here with a later version of kubernetes client API that it might work. I headed over to main maven repository and downloaded following jar files.

https://repo1.maven.org/maven2/io/fabric8/kubernetes-model/4.6.4/kubernetes-client-4.6.4.jar
https://repo1.maven.org/maven2/io/fabric8/kubernetes-model/4.6.4/kubernetes-model-4.6.4.jar
https://repo1.maven.org/maven2/io/fabric8/kubernetes-model/4.6.4/kubernetes-model-common-4.6.4.jar

After that you only need the following line in your Dockerfile to remove the older version and replace with this version. Note lib-override is where I copied the jar files downloaded above.

RUN rm /opt/spark/jars/kubernetes*4.1.2.jar && cp /app/lib-override/* /opt/spark/jars

This solved the above issue and luckily there were no further errors. This is probably not the best way to do it, but I didn’t want to upgrade the spark version without further testing. Your mileage may vary!

Written by Francois Fernando, a software craftsman and tinkerer.