跳至主要内容

Spark No suitable driver found for jdbc

 The famous driver not found issue

I was request to draw a branch of data out of big data platform to MySQL recently. After some carefully consideration, I decide to code the logic in spark with the DataFrameWriter.write.jdbc() interface.

Everything goes well until the integration test, in which I have to install my spark app for a real run.

java.sql.SQLException: No suitable driver found for jdbc:mysql://bla:bla/bla

Background1: I do follow the spark jdbc guide

Actually, I did the simulation by follow the DATABRICKS: Connecting to SQL Databases using JDBC on my own zeppline notebook. And everything went well.

Background2: I do follow the MySQL jdbc guide

I did follow the official MySQL Connector/j: 6.1 Connecting to MySQL Using the JDBC DriverManager Interface coding sample.

// The newInstance() call is a work around for some
// broken Java implementations

Class.forName("com.mysql.jdbc.Driver").newInstance();

Background3: I do attach the jar dependency

I did attach the connector/j jar in spark-submit –jars argument

Attampt 1: Use recent version of MySQL connector/j

As suggested by many stackoverflow user, it’s a pre-jdbc 4.0 issue, only jdbc driver complains with jdbc 4.0 can be discovered and registerd automatically.

I upgraded from connector/j 5.1.18 to 5.1.38 and later. The Exception only goes away on ******some****** of our cluster running Cloudera 5.11 on JDK8 and JRE8.

On clusters with Cloudera 5.11 and official cloudera JDK7 the issue remains.

Attempt 2: Explicitly set the driver class in jdbc properties

After a lot of search, I find the final workaround for the issue. AS TITLE.

With the following code snip, my app now works on any Cloudera supported JDK with any version of MySQL connector/j.

// explicitly set the driver, or, you will get the mysterious 'java.sql.SQLException: No suitable driver' on JDK 1.7.0_67
connectionProperties.put("driver", "com.mysql.jdbc.Driver")

Lesson learned

Explicit is not bad!!

评论

此博客中的热门博文

Eglot and before/after-save-hook and use-package

In Emacs, when you try to automate some actions during every save action, you will surely get to the before-save-hook and the after-save-hook. Simply adding something like gofmt-before-save to before-save-hook will save you tons of time to do the go-fmt. And then, I meet eglot, and gopls will also save me tons of time doing googling and api documentation navigation. But eglot-ensure is not very friendly to the good old ways of how after-save-hooks were designed to work. It makes the before/after-save-hook a buffer local variable and it does not inherit the variable's global value. So, to make before/after-save-hook work again, experts start to adding hooks to major mode specific hooks like this: emacs.md - Go (opensource.google) """ ;; Optional: install eglot-format-buffer as a save hook. ;; The depth of -10 places this before eglot's willSave notification, ;; so that that notification reports the actual contents that will be saved. (defu...

XEmacs 21.5 beta 35 "kohlrabi" has been released.

If you are an old XEmacs user, you may feel happy to see this from https://www.xemacs.org/.    After ten years, XEmacs released a new version 21.5. So there's still many people cares about XEmacs. The XEmacs' source repo have been moved from altassian Bitbucket to https://heptapod.net/. As Bitbucket have been dropped Mercurial support many years ago.