Spark No suitable driver found for jdbc

 The famous driver not found issue

I was request to draw a branch of data out of big data platform to MySQL recently. After some carefully consideration, I decide to code the logic in spark with the DataFrameWriter.write.jdbc() interface.

Everything goes well until the integration test, in which I have to install my spark app for a real run.

java.sql.SQLException: No suitable driver found for jdbc:mysql://bla:bla/bla

Background1: I do follow the spark jdbc guide

Actually, I did the simulation by follow the DATABRICKS: Connecting to SQL Databases using JDBC on my own zeppline notebook. And everything went well.

Background2: I do follow the MySQL jdbc guide

I did follow the official MySQL Connector/j: 6.1 Connecting to MySQL Using the JDBC DriverManager Interface coding sample.

// The newInstance() call is a work around for some
// broken Java implementations

Class.forName("com.mysql.jdbc.Driver").newInstance();

Background3: I do attach the jar dependency

I did attach the connector/j jar in spark-submit –jars argument

Attampt 1: Use recent version of MySQL connector/j

As suggested by many stackoverflow user, it’s a pre-jdbc 4.0 issue, only jdbc driver complains with jdbc 4.0 can be discovered and registerd automatically.

I upgraded from connector/j 5.1.18 to 5.1.38 and later. The Exception only goes away on ******some****** of our cluster running Cloudera 5.11 on JDK8 and JRE8.

On clusters with Cloudera 5.11 and official cloudera JDK7 the issue remains.

Attempt 2: Explicitly set the driver class in jdbc properties

After a lot of search, I find the final workaround for the issue. AS TITLE.

With the following code snip, my app now works on any Cloudera supported JDK with any version of MySQL connector/j.

// explicitly set the driver, or, you will get the mysterious 'java.sql.SQLException: No suitable driver' on JDK 1.7.0_67
connectionProperties.put("driver", "com.mysql.jdbc.Driver")

Lesson learned

Explicit is not bad!!