java – BigQueryException: Remote host terminated the handshake

I am running a pyspark application locally, the application reads a table in BigQuery. Below are the details,

Spark submit command:

spark-submit --name "sampleApp" --master local[2] --deploy-mode client --packages com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.17.3 --verbose main.py

Pyspark code

    df = spark.read 
        .format("bigquery") 
        .load("projectName.datasetName.tableName")

I’ve set the environment variable to point to the path of the Secret JSON file in local.

GOOGLE_APPLICATION_CREDENTIALS

Pyspark version is

pyspark==3.1.3

I’ve also tried using “com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.25.0” package in spark-submit command.

I’m connected to VPN and I could see from the logs that all the dependencies are successfully downloaded, but I get the below error after submitting the spark job

py4j.protocol.Py4JJavaError: An error occurred while calling o27.load. :
com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryException:
Remote host terminated the handshake
at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:106) at com.google.cloud.spark.bigquery.repackaged.com. google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:268) at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryImpl$17.call(Big.jaQueryIm) 669) at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryImpl$17.call(BigQueryImpl.java:666) at com.google.cloud.spark.bigquery.repackaged.com.google .api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105) at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.RetryHelper.run(RetryHelper.java:76) at com.google .cloud.spark.bigquery.repackaged.com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryImpl.getTable( BigQueryImpl.java:665) at com.google.cloud.spark.bigquery.BigQuer yRelationProvider.createRelationInternal(BigQueryRelationProvider.scala:75) at com.google.cloud.spark.bigquery.BigQueryRelationProvider.createRelation(BigQueryRelationProvider.scala:45) at org.apache.spark.sql.executionDreataResourceSdataS scala:355) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325) at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307) at scala. .getOrElse(Option.scala:189) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:239) at sun. reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingjacessorIthompl.43). Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection. ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.exeja(CallCommand.execute(CallCommand) :79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:750) Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1575) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1405) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord13.pl.SLSocketImpl. ) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) at sun.net.www.protocol.https. AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197) at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:167) at com.google.cloud.spark.googleabigquery.repack.repack client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:148) at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84 ) at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) at com.google.cloud.spark.bigquery.repackaged.com.g oogle.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541) at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.services.AbstractGoogleClientRequestRequestsGoogleExe.GooglecuteRequestClient .java:474) at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591) at com.google.cloud.spark.bigquery .repackaged.com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:266) … 25 more Caused by: java.io.EOFException: SSL peer shut down incorrectly at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:167) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109) at sun.security .ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1397) … 37 more

How can I resolve this error?

Leave a Comment