跳至主要内容

Cloudera 5.11.x Spark action on oozie failed to access hive table

Spark workflow fails for not able to access hive table

This is an odd issue. With the same spark program, it will fail to access hive table if scheduled by Ozzie, but runs well if run manually by using spark-submit.

Solution

Add hive-site.xml to the spark workflow to make sure hive context is correctly initialized.

  • Put the hive-site.xml into hdfs

    On one of the cluster node, find the hive configuration XML ‘hive-site.xml’ from /etc/hive/conf. Copy it to somewhere on hdfs.

  • Add the hive-site.xml as one of the “FILES” of spark workflow

    In Hue workflow editor, click the plus sign on “FILES” to add a new “FILE” element. Write the corresponding hdfs path of just copied ‘hive-site.xml’.

 

标签：

评论

发表评论

此博客中的热门博文

XEmacs 21.5 beta 35 "kohlrabi" has been released.

If you are an old XEmacs user, you may feel happy to see this from https://www.xemacs.org/.    After ten years, XEmacs released a new version 21.5. So there's still many people cares about XEmacs. The XEmacs' source repo have been moved from altassian Bitbucket to https://heptapod.net/. As Bitbucket have been dropped Mercurial support many years ago.
发表评论
阅读全文

Fido-mode

Today, I've just discovered the Fido mode, a modified `icomplete` minor mode. `icomplete` used to be one of my favorite mode days back to my XEmacs days. Pros: It is way fast. It is much more smart in terms of find the complete candidate that you want mostly.  Cons: It does not integrate well enough with tramp yet.    `fido-mode` is a core package of Emacs 27+, and there's a vertical version available after Emacs 28+ named `fido-vertical-mode`. UPDATE: I am back with ido now
发表评论
阅读全文