鸡毛蒜皮

I want to build the bfg tool In my previous blog I talked about bfg the bfg-repo-cleaner . It is a fast repo maintain tool written in scala. The developer didn't provide any prebuilt binary downloads. So, clone and built by myself. SBT do not work well the maven way After some simple git clone. Backed with some maven knowledge and knowing that sbt is kind of same tool. I thought like a fool that a sdk install sbt followed by sbt build command will built a beautiful jar in target directory as maven package do. Sbt version mismatch Quickly, I learn the first lesson from sbt: sbt will not build the old project with latest sbt version, it will always try to build the project with declared version. If the version is not available locally, SBT will try to download it until successes with the downloading. sbt will download with project declared sbt version if not already installed. sbt will download the project declared scala version if not already installed. s...

Problem: dataframe are not shared? As of version Zepplin(0.7.0), Spark dataframe are not shared between %pyspark (python) and %spark (scala) session. Solution: exchange by using temporary table do the following #%pyspark somedf.registerTempTable("somedftable") and then rebuild the DataFrame in scala session //%scala val somedf = sqlContext.table("somedftable") z.show(somedf.limit(20))  

 Lesson learn Scala programming I got some Scala to write in a recent spark related project. Specially, the Scala UDF (aka. user defined function) to be used in spark application. I got a feel that Try, Success, Failure is not very useful until being applied to the For Comprehensions for automatically unpacking. Refer to the following code example (inspired by StackOverFlow Post ) import java.sql.Timestamp import org.apache.spark.sql.functions.udf import scala.util.{Try, Success, Failure} // combine time_date and time_time for a timestamp in UTC. val time_to_ts: ((String, String) => Option[Timestamp]) = (time_date, time_time) => { val time_time_a = time_time.split(':') val ts = for ( y <- Try(time_date.take(4).toInt); // toInt exception m <- Try(time_date.take(6).takeRight(2).toInt); // toInt exception d <- Try(time_date.take(8).takeRight(2).toInt); // toInt exception tsymd <- Try(Timestamp.valueOf(s"$y-$m-$d 0:0:...