A comprehensive guide on deploying Apache Spark on Kubernetes, exploring challenges, architecture decisions, and implementation details for a cloud-agnostic solution