给出以下示例： d = [{'asset': '2', 'ts': 6, 'B':'123','C':'234'}, {'asset': '1', 'ts': 5, 'C.1':'999', 'B':'888','F':'999'}] df = spark.createDataFrame(d) df.show(truncate=False) +---+----+-----+---+----+----+ |B |C |asset|ts |C.1 |F | +---+----+----...

我有一个2个数据帧，如下所示： 第一个df1 TEST_schema = StructType([StructField("description", StringType(), True),\ StructField("date", StringType(), True)\ ]) TEST_data = [('START',20200622),('END',20201018)] rdd3 = sc.pa...

I am a beginner in Kafka and I am trying to read the data from a Kafka topic using spark in scala. 我的主要功能是： def main(args : Array[String]) : Unit = { val spark = SparkSession .builder() .appName("testKafka") .master("local[*]") .getOrCreat...

从类似日期（年）的列中，我得到了这种类型的值： +--------------------+------------+ | CUSTOMER_ID|yearSelected| +--------------------+------------+ |1 | 2010| |2 | 1992| |3 | 1996| |4 ...

This question already has answers here: (null) entry in command string exception in saveAsTextFile() on Pyspark ...

To understand this problem, imagine that a group of people, at time t1, are in a certain location L. You are interested in knowing where all those people were at time t0. 我在这里有一个示例代码来引导练习： row = Row("user_id", "start", "location_id") df = spark.sparkContex...

我有一个用例，可以向timestamp列中添加一些值并从中得出日期。我正在尝试使用from_unix和unix_timestamp函数。但是它以不同的格式显示了不同的值。 select from_unixtime('1577582583', 'YYYY-MM-dd'), from_unixtime('1577582583', 'yyyy-MM-dd'); from_unixtime(CAST(1577582583 AS BIGINT), YYYY-MM-dd) from_unixtime(CAST(...