跳至主要内容

Exchange data between zeppelin pyspark and spark session

Problem: dataframe are not shared?

As of version Zepplin(0.7.0), Spark dataframe are not shared between %pyspark (python) and %spark (scala) session.

Solution: exchange by using temporary table

do the following

#%pyspark

somedf.registerTempTable("somedftable")

and then rebuild the DataFrame in scala session

//%scala

val somedf = sqlContext.table("somedftable")

z.show(somedf.limit(20))

 

评论

此博客中的热门博文

XEmacs 21.5 beta 35 "kohlrabi" has been released.

If you are an old XEmacs user, you may feel happy to see this from https://www.xemacs.org/.    After ten years, XEmacs released a new version 21.5. So there's still many people cares about XEmacs. The XEmacs' source repo have been moved from altassian Bitbucket to https://heptapod.net/. As Bitbucket have been dropped Mercurial support many years ago.

Fido-mode

Today, I've just discovered the Fido mode, a modified `icomplete` minor mode. `icomplete` used to be one of my favorite mode days back to my XEmacs days. Pros: It is way fast. It is much more smart in terms of find the complete candidate that you want mostly.  Cons: It does not integrate well enough with tramp yet.    `fido-mode` is a core package of Emacs 27+, and there's a vertical version available after Emacs 28+ named `fido-vertical-mode`. UPDATE: I am back with ido now