pyspark – java.lang.ClassFormatError: Unknown constant tag 83 in class file org/apache/spark/ml/param/DoubleArrayParam Exception when uploading SparkModel to S3

We are running a Spark Job in our kubernetes cluster, in which we are trying to save a Spark PipelineModel to S3. When submitting the job with only 1 executor, everything works fine, but when using 2 or more executors the job fails with the following Error: 22/07/01 07:07:48 ERROR Instrumentation: org.apache.spark.SparkException: Job aborted due … Read more