鸡毛蒜皮

 Hunspell failed with default setup With my shining new emacs26.2 setup, I am happy until trying to `M-$’ on a word. I got following error message, it used to be work by just set ispell program to hunspell. Starting new Ispell process hunspell with default dictionary... split-string: Wrong type argument: stringp, nil After some check in my customization setting. nothing found. install a copy of hunspell dictionary At first I think it was caused by no hunspell dictionary. I downloaded en US and en CA hunspell dictionary from wordlist.sf.net project. After put the .aff and .dic files into C:directory I’ve also made a symbolic link from en US ones by running the `mklink’ tool: mklink default.aff en_US.aff mklink default.dic en_US.dic Err… hunspell does not work, still. Time to read code Dig into ispell.el I suprisingly find there’re lots of hunspell specific logic. Originally I was thinking hunspell support involves no elisp work. Acturally ispell did many work to de...

 The famous driver not found issue I was request to draw a branch of data out of big data platform to MySQL recently. After some carefully consideration, I decide to code the logic in spark with the DataFrameWriter.write.jdbc() interface. Everything goes well until the integration test, in which I have to install my spark app for a real run. java.sql.SQLException: No suitable driver found for jdbc:mysql://bla:bla/bla Background1: I do follow the spark jdbc guide Actually, I did the simulation by follow the DATABRICKS: Connecting to SQL Databases using JDBC on my own zeppline notebook. And everything went well. Background2: I do follow the MySQL jdbc guide I did follow the official MySQL Connector/j: 6.1 Connecting to MySQL Using the JDBC DriverManager Interface coding sample. // The newInstance() call is a work around for some // broken Java implementations Class.forName("com.mysql.jdbc.Driver").newInstance(); Background3: I do attach the jar dependency I did a...

Ever want to export periodically generated hive data to RDBMS? There may be several solutions available, depends on the data size and update interval. But if you decide to use sqoop1 to export the ******Hive table******, don’t follow those handcraft solution suggestions, try the sqoop1 internal one. The handcraft solution: copy, export, delete As title, this kind of solution mainly consist of 3 steps. Copy data out from warehouse to general HDFS path Run sqoop command against the copied HDFS path Delete the copied HDFS path The sqoop1 internal solution: HCatalog arguments Run sqoop-export –help and pay attention to the ****** HCatalog arguments****** section HCatalog arguments: --hcatalog-database <arg> HCatalog database name --hcatalog-home <hdir> Override $HCAT_HOME --hcatalog-partition-keys <partition-key> Sets the partition keys to use when importin...

Problem: dataframe are not shared? As of version Zepplin(0.7.0), Spark dataframe are not shared between %pyspark (python) and %spark (scala) session. Solution: exchange by using temporary table do the following #%pyspark somedf.registerTempTable("somedftable") and then rebuild the DataFrame in scala session //%scala val somedf = sqlContext.table("somedftable") z.show(somedf.limit(20))  

No TypeTag available compiling… Following code works whell in zeppelin section val textRdd = sc.textFile("hdfs://nameservice1/user/myname/bank/bank.csv") case class TextLine(lineText: String) val modelDates = textRdd.map( s => TextLine(s.trim)).toDF() modelDates.sort(col("lineText").desc).as[String].first() But if I make a function from it, err!! def maxValueIn(hdfsPath: String) = { val textRdd = sc.textFile(hdfsPath) case class TextLine(lineText: String) val modelDates = textRdd.map( s => TextLine(s.trim)).toDF() modelDates.sort(col("lineText").desc).as[String].first() } Solution: Move the case class out of def!!! As described in the stack overflow question , move the case class out of the method, the code finally compiles. import org.apache.spark.SparkContext import org.apache.spark.sql.hive.HiveContext import org.apache.spark.sql.functions.col trait HdfsTextFile { val spark: SparkContext val sqlContext: HiveContext impo...

Spark workflow fails for not able to access hive table This is an odd issue. With the same spark program, it will fail to access hive table if scheduled by Ozzie, but runs well if run manually by using spark-submit. Solution Add hive-site.xml to the spark workflow to make sure hive context is correctly initialized. Put the hive-site.xml into hdfs On one of the cluster node, find the hive configuration XML ‘hive-site.xml’ from /etc/hive/conf. Copy it to somewhere on hdfs. Add the hive-site.xml as one of the “FILES” of spark workflow In Hue workflow editor, click the plus sign on “FILES” to add a new “FILE” element. Write the corresponding hdfs path of just copied ‘hive-site.xml’.  

The beautiful, open source, with ligatures, font Fira Code Some month ago I see someone talking about the best fonts and themes for their Intellij IDEA setup. One of the setups is really good, it was my first time to know that symbols such as `<=’ or `->’ can be display as beautiful arrows at runtime by supported software and a supported font. The font that mentioned was Fira Code Install FiraCode and fonts mess up in various programs I do like the beautiful ligatures. And, `choco search fira’ shows that the font can be easily installed by Chocolatey, Great!!! Anyway, things are not as good as I expected. Fira code fonts seems not work well in my Windows 7 ClearType environment. It shows up hideous in my Emacs buffer, even worse, it will mess up totally in J2SE Swing program if subpixel hint is turned on. Recovery method I tried several recovery methods, they are either no help or fail back to old situation after one or two cycles of Windows reboot. Remove/reinsta...