跳至主要内容

Spark No suitable driver found for jdbc

 The famous driver not found issue

I was request to draw a branch of data out of big data platform to MySQL recently. After some carefully consideration, I decide to code the logic in spark with the DataFrameWriter.write.jdbc() interface.

Everything goes well until the integration test, in which I have to install my spark app for a real run.

java.sql.SQLException: No suitable driver found for jdbc:mysql://bla:bla/bla

Background1: I do follow the spark jdbc guide

Actually, I did the simulation by follow the DATABRICKS: Connecting to SQL Databases using JDBC on my own zeppline notebook. And everything went well.

Background2: I do follow the MySQL jdbc guide

I did follow the official MySQL Connector/j: 6.1 Connecting to MySQL Using the JDBC DriverManager Interface coding sample.

// The newInstance() call is a work around for some
// broken Java implementations

Class.forName("com.mysql.jdbc.Driver").newInstance();

Background3: I do attach the jar dependency

I did attach the connector/j jar in spark-submit –jars argument

Attampt 1: Use recent version of MySQL connector/j

As suggested by many stackoverflow user, it’s a pre-jdbc 4.0 issue, only jdbc driver complains with jdbc 4.0 can be discovered and registerd automatically.

I upgraded from connector/j 5.1.18 to 5.1.38 and later. The Exception only goes away on ******some****** of our cluster running Cloudera 5.11 on JDK8 and JRE8.

On clusters with Cloudera 5.11 and official cloudera JDK7 the issue remains.

Attempt 2: Explicitly set the driver class in jdbc properties

After a lot of search, I find the final workaround for the issue. AS TITLE.

With the following code snip, my app now works on any Cloudera supported JDK with any version of MySQL connector/j.

// explicitly set the driver, or, you will get the mysterious 'java.sql.SQLException: No suitable driver' on JDK 1.7.0_67
connectionProperties.put("driver", "com.mysql.jdbc.Driver")

Lesson learned

Explicit is not bad!!

标签：

评论

发表评论

此博客中的热门博文

Eglot and before/after-save-hook and use-package

In Emacs, when you try to automate some actions during every save action, you will surely get to the before-save-hook and the after-save-hook. Simply adding something like gofmt-before-save to before-save-hook will save you tons of time to do the go-fmt. And then, I meet eglot, and gopls will also save me tons of time doing googling and api documentation navigation. But eglot-ensure is not very friendly to the good old ways of how after-save-hooks were designed to work. It makes the before/after-save-hook a buffer local variable and it does not inherit the variable's global value. So, to make before/after-save-hook work again, experts start to adding hooks to major mode specific hooks like this: emacs.md - Go (opensource.google) """ ;; Optional: install eglot-format-buffer as a save hook. ;; The depth of -10 places this before eglot's willSave notification, ;; so that that notification reports the actual contents that will be saved. (defu...
发表评论
阅读全文

Use MobaDiff with git difftool

Recently there's an activity in IT that forces the deletion of all unauthorized softwares from all work machines. Unfortunately, kdiff3 is one in the list. As it is generally okay to use vimdiff as an alternative for kdiff3, A gui tool is better suited for desktop workflows. Known that MobaXterm is shipping a gui diff tool named MobaDiff. But it only appears in the windows right click context menu. Find the real application name takes me some time to search in the windows registry. "MobaRTE.exe", which is the one invoked by HKCR\*\shell\MobaDiff. And it was invoked with "-contextdiff" switch to show MobaDiff UI, while when the switch is "-contextedit" it shows MobaTextEditor. Too bad that the "-contextdiff" switch do not support pre-image post-image as other diff tool did, which effectively made it unable to be used as a command line diff utility. Also MobaTech did not mention anything in their document of this Mob...
发表评论
阅读全文