Inside the driver container — process, JVM and REPL An architecture diagram generated by Archify. dbr_entry_point.py · real ENTRYPOINT · Architecture component · PID ~1 dbr_entry_point.py real ENTRYPOINT PID ~1 Spark Master · OSS standalone · :7077 · Architecture component Spark Master OSS standalone · :7077 ChauffeurDaemon · :6060 · :7071 · :7788 · Architecture component · only public edge ChauffeurDaemon :6060 · :7071 · :7788 only public edge DriverDaemonMonitor · inside Chauffeur JVM · Architecture component DriverDaemonMonitor inside Chauffeur JVM DriverDaemon · composition root · DriverDaemon JVM — one process · holds no SparkContext DriverDaemon composition root holds no SparkContext ReadinessAwareDriverCorral · gate isDefined/isReady · DriverDaemon JVM — one process ReadinessAwareDriverCorral gate isDefined/isReady DriverCorral · REPL dispatcher · DriverDaemon JVM — one process DriverCorral REPL dispatcher DatabricksILoop$ · volatile singleton · DriverDaemon JVM — one process DatabricksILoop$ volatile singleton SharedDriverContext · wraps SparkContext · DriverDaemon JVM — one process · one per cluster SharedDriverContext wraps SparkContext one per cluster DriverLocal ×4 · Scala · Python · SQL · R · DriverDaemon JVM — one process · own SparkAdapter DriverLocal ×4 Scala · Python · SQL · R own SparkAdapter DriverILoop · extends SparkILoop · DriverDaemon JVM — one process · IN-PROCESS DriverILoop extends SparkILoop IN-PROCESS ipykernel · separate process · Architecture component ipykernel separate process Almond kernel · isolated Scala kernel · Architecture component Almond kernel isolated Scala kernel start_spark_master.sh start_chauffeur.sh start_driver.sh · child process same JVM poll 4s · jstack/jmap new DriverCorral publish · setReady() get() when ready sharedEnvContext() lazy getOrCreate · appName "Databricks Shell" allocate REPL driverContext() in-process ZMQ + HMAC isolated Scala DriverDaemon JVM — one process Legend REPL Driver component Global singleton Orchestrator process Monitor / gate Standalone process