Welsige/scala_remove_duplicates.scala

Created September 15, 2020 00:22

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Select an option

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/Welsige/85e762dc1f978a0e41f73d03990611e3.js"></script>
Save Welsige/85e762dc1f978a0e41f73d03990611e3 to your computer and use it in GitHub Desktop.

Download ZIP

Scala Spark remove duplicates completely

Raw

scala_remove_duplicates.scala

val df02 = df01.withColumn("cnt", count("*").over(Window.partitionBy($"id"))).where($"cnt"===1).drop($"cnt")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment