跳至主要内容

Mysterious sqoop1 export options

Ever want to export periodically generated hive data to RDBMS?

There may be several solutions available, depends on the data size and update interval. But if you decide to use sqoop1 to export the ******Hive table******, don’t follow those handcraft solution suggestions, try the sqoop1 internal one.

The handcraft solution: copy, export, delete

As title, this kind of solution mainly consist of 3 steps.

  1. Copy data out from warehouse to general HDFS path
  2. Run sqoop command against the copied HDFS path
  3. Delete the copied HDFS path

The sqoop1 internal solution: HCatalog arguments

Run sqoop-export –help and pay attention to the ******HCatalog arguments****** section

HCatalog arguments:
--hcatalog-database <arg>                        HCatalog database name
--hcatalog-home <hdir>                           Override $HCAT_HOME
--hcatalog-partition-keys <partition-key>        Sets the partition
                         keys to use when
                         importing to hive
--hcatalog-partition-values <partition-value>    Sets the partition
                         values to use when
                         importing to hive
--hcatalog-table <arg>                           HCatalog table name
--hive-home <dir>                                Override $HIVE_HOME
--hive-partition-key <partition-key>             Sets the partition key
                         to use when importing
                         to hive
--hive-partition-value <partition-value>         Sets the partition
                         value to use when
                         importing to hive
--map-column-hive <arg>                          Override mapping for
                         specific column to
                         hive types.

Note that –hcatalog-partition-keys and –hcatalog-partition-values and –hive-partition-key and –hive-partition-value.

Try to specify partitions and partition values as comma separated list for pural ones.

Even these arguments’ help string is wrongly stated they are ******import****** options, they did the partition filtering job very good when exporting.

标签：

评论

发表评论

此博客中的热门博文

Eglot and before/after-save-hook and use-package

In Emacs, when you try to automate some actions during every save action, you will surely get to the before-save-hook and the after-save-hook. Simply adding something like gofmt-before-save to before-save-hook will save you tons of time to do the go-fmt. And then, I meet eglot, and gopls will also save me tons of time doing googling and api documentation navigation. But eglot-ensure is not very friendly to the good old ways of how after-save-hooks were designed to work. It makes the before/after-save-hook a buffer local variable and it does not inherit the variable's global value. So, to make before/after-save-hook work again, experts start to adding hooks to major mode specific hooks like this: emacs.md - Go (opensource.google) """ ;; Optional: install eglot-format-buffer as a save hook. ;; The depth of -10 places this before eglot's willSave notification, ;; so that that notification reports the actual contents that will be saved. (defu...
发表评论
阅读全文

Use MobaDiff with git difftool

Recently there's an activity in IT that forces the deletion of all unauthorized softwares from all work machines. Unfortunately, kdiff3 is one in the list. As it is generally okay to use vimdiff as an alternative for kdiff3, A gui tool is better suited for desktop workflows. Known that MobaXterm is shipping a gui diff tool named MobaDiff. But it only appears in the windows right click context menu. Find the real application name takes me some time to search in the windows registry. "MobaRTE.exe", which is the one invoked by HKCR\*\shell\MobaDiff. And it was invoked with "-contextdiff" switch to show MobaDiff UI, while when the switch is "-contextedit" it shows MobaTextEditor. Too bad that the "-contextdiff" switch do not support pre-image post-image as other diff tool did, which effectively made it unable to be used as a command line diff utility. Also MobaTech did not mention anything in their document of this Mob...
发表评论
阅读全文