跳至主要内容

Cloudera 5.11.x Spark action on oozie failed to access hive table

Spark workflow fails for not able to access hive table

This is an odd issue. With the same spark program, it will fail to access hive table if scheduled by Ozzie, but runs well if run manually by using spark-submit.

Solution

Add hive-site.xml to the spark workflow to make sure hive context is correctly initialized.

  • Put the hive-site.xml into hdfs

    On one of the cluster node, find the hive configuration XML ‘hive-site.xml’ from /etc/hive/conf. Copy it to somewhere on hdfs.

  • Add the hive-site.xml as one of the “FILES” of spark workflow

    In Hue workflow editor, click the plus sign on “FILES” to add a new “FILE” element. Write the corresponding hdfs path of just copied ‘hive-site.xml’.

 

标签：

评论

发表评论

此博客中的热门博文

Eglot and before/after-save-hook and use-package

In Emacs, when you try to automate some actions during every save action, you will surely get to the before-save-hook and the after-save-hook. Simply adding something like gofmt-before-save to before-save-hook will save you tons of time to do the go-fmt. And then, I meet eglot, and gopls will also save me tons of time doing googling and api documentation navigation. But eglot-ensure is not very friendly to the good old ways of how after-save-hooks were designed to work. It makes the before/after-save-hook a buffer local variable and it does not inherit the variable's global value. So, to make before/after-save-hook work again, experts start to adding hooks to major mode specific hooks like this: emacs.md - Go (opensource.google) """ ;; Optional: install eglot-format-buffer as a save hook. ;; The depth of -10 places this before eglot's willSave notification, ;; so that that notification reports the actual contents that will be saved. (defu...
发表评论
阅读全文

XEmacs 21.5 beta 35 "kohlrabi" has been released.

If you are an old XEmacs user, you may feel happy to see this from https://www.xemacs.org/.    After ten years, XEmacs released a new version 21.5. So there's still many people cares about XEmacs. The XEmacs' source repo have been moved from altassian Bitbucket to https://heptapod.net/. As Bitbucket have been dropped Mercurial support many years ago.
发表评论
阅读全文