I've been caught out by doing date comparisons in Spark SQL using string yyyy-mm-dd against string yyyy-mm-dd 00:00:00. It's easy to think that the trailing 00:00:00 will have no impact on date comparisons but it does! Lesson learnt: use substr(date, 1,10)