java – BigQueryException: Remote host terminated the handshake

I am running a pyspark application locally, the application reads a table in BigQuery. Below are the details,

Spark submit command:

spark-submit --name "sampleApp" --master local[2] --deploy-mode client --packages --verbose

Pyspark code

    df = 

I’ve set the environment variable to point to the path of the Secret JSON file in local.


Pyspark version is


I’ve also tried using “” package in spark-submit command.

I’m connected to VPN and I could see from the logs that all the dependencies are successfully downloaded, but I get the below error after submitting the spark job

py4j.protocol.Py4JJavaError: An error occurred while calling o27.load. :
Remote host terminated the handshake
at at at$ 669) at$ at .api.gax.retrying.DirectRetryingExecutor.submit( at at at at yRelationProvider.createRelationInternal(BigQueryRelationProvider.scala:75) at at org.apache.spark.sql.executionDreataResourceSdataS scala:355) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325) at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307) at scala. .getOrElse(Option.scala:189) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:239) at sun. reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke( at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingjacessorIthompl.43). at py4j.reflection.MethodInvoker.invoke( at py4j.reflection. ReflectionEngine.invoke( at py4j.Gateway.invoke( at py4j.commands.AbstractCommand.invokeMethod( at py4j.commands.CallCommand.exeja(CallCommand.execute(CallCommand) :79) at at Caused by: Remote host terminated the handshake
at at at ) at at at AbstractDelegateHttpsURLConnection.connect( at at client.http.javanet.NetHttpRequest.execute( at ) at at at .java:474) at at … 25 more Caused by: SSL peer shut down incorrectly at at at .ssl.SSLSocketImpl.decode( … 37 more

How can I resolve this error?

Leave a Comment