Background

Code assessment and static evaluation instruments are normal practices within the Software program Growth Lifecycle (SDLC). Static evaluation instruments assist builders discover code high quality points, guarantee adherence to coding requirements, and determine doable safety points. In an interactive pocket book surroundings the place customers run ad-hoc instructions iteratively, there is not a effectively outlined sample for making use of these normal SDLC practices. Nevertheless, as customers could be working with extremely delicate information, we nonetheless wish to monitor that correct safety finest practices are being utilized simply as we’d for automated manufacturing pipelines.

Notice that given the character of the Databricks platform, not all widespread software program safety points are related. Utilizing OWASP Prime 10 as a place to begin, points resembling cross-site scripting or different injection assaults do not make sense since customers aren’t working net purposes of their notebooks.

Handbook code assessment or “spot-checks” of pocket book code is feasible, however not scalable since there could also be dozens to lots of of customers on the platform working 1000’s of instructions per day. We’d like a solution to automate these checks to seek out essentially the most essential points. The introduction of Databricks verbose pocket book audit logs permits us to observe instructions run by customers and apply the detections we would like in a scalable, automated trend. On this doc, we share one instance of utilizing a Python static evaluation instrument to observe for widespread safety points resembling mishandling credentials and secrets and techniques. To be clear, automated static evaluation instruments assist us scale these kinds of checks however aren’t a alternative for correct safety controls resembling information loss protections and entry controls.

On this article, we is not going to focus on how one can configure audit logs, as that’s lined in our documentation (AWS, GCP, Azure). For AWS, we have additionally beforehand revealed a weblog put up with instance code to do that.

Monitoring pocket book command logs

The workspace audit log documentation contains particulars on enabling verbose audit logs and the extra occasions provided for pocket book instructions. As soon as the occasions are included in your audit logs you may start monitoring them and making use of some detections. Whereas we will definitely apply easy text-based comparisons utilizing common expressions or trying to find particular key phrases within the particular person instructions, this has a number of limitations. Specifically, easy textual content searches will miss management and information circulate. For instance, if a person assigns a credential from a secret scope to a variable in a single command, then in a while writes that worth to a file or logs it in one other command, a easy textual content search will likely be unable to detect it.

Within the instance beneath, the person reads JDBC credentials from Secret Scopes and makes an attempt to load a DataFrame from the database. Within the occasion of an error, the connection string with embedded credentials is written to output. It is a dangerous apply as these credentials will now leak into logs. A easy textual content search wouldn’t be capable of reliably hint the password from the supply to the “sink” which is printing to output.

db_username = dbutils.secrets and techniques.get("db", "username")
db_password = dbutils.secrets and techniques.get("db", "password") # supply of delicate information
# worth will get assigned to a brand new variable
db_url = f"jdbc:mysql://{host}/{schema}?person={db_username}&password={db_password}"

strive:
    df = (spark.learn.format("jdbc")
             .possibility("url", db_url)
             .possibility("dbtable", desk)
             .load())
besides:
    print("Error connecting to JDBC datasource")
    print(db_url) # potential leak of delicate information

Nevertheless, a static evaluation instrument with management and information circulate evaluation can do that simply and reliably to alert us to the potential danger. An open supply venture referred to as Pysa, part of the Pyre venture, gives Python static evaluation with the power to outline customized guidelines for the forms of points we wish to detect. Pyre is a really succesful instrument with a number of options, we is not going to go into all the small print on this doc. We advocate that you just learn the documentation and observe the tutorials for extra data. You can even use different instruments should you favor, together with different languages resembling R or Scala. The method defined on this doc ought to apply to different instruments and languages.

Earlier than working the static evaluation we have to group the instructions run within the pocket book so no matter instrument we’re utilizing can construct a correct name graph. It’s because we wish to preserve the context of what instructions had been run and in what order so the code may be analyzed correctly. We do that by ordering and sessionizing the instructions run for every pocket book. The audit logs give us the notebook_id, command_id, command_text, and a timestamp. With that we will order and group the instructions executed inside a session. We’ll think about the beginning of a session when a pocket book is first connected to a cluster till the cluster terminates or the pocket book is indifferent. As soon as the instructions are grouped collectively and ordered, we will go the code to the static evaluation instrument.

# get all profitable pocket book instructions for the time interval
instructions = (spark.learn.desk("log_data.workspace_audit_logs")
            .filter(f"serviceName = 'pocket book' and actionName in ('runCommand', 'attachNotebook') and date >= current_date() - interval {lookback_days} days")
            .filter("requestParams.path shouldn't be null or requestParams.commandText not like '%%'"))

# sessionize primarily based on connect occasions
sessionized = (instructions
               .withColumn("notebook_path", F.when(F.col("actionName") == "attachNotebook", F.col("requestParams.path")).in any other case(None))
               .withColumn("session_started", F.col("actionName") == "attachNotebook")
               .withColumn("session_id", F.sum(F.when(F.col("session_started"), 1).in any other case(0)).over(Window.partitionBy("requestParams.notebookId").orderBy("timestamp")))
               .withColumn("notebook_path", F.first("notebook_path").over(Window.partitionBy("session_id", "requestParams.notebookId").orderBy("timestamp"))))

Most instruments count on the code to scan to be recordsdata on disk. We do that by taking the instructions we sessionized then writing them to non permanent recordsdata which are scanned by Pyre. For Pyre, we additionally have to configure sure gadgets resembling the principles we wish to apply and describing the supply and sink of delicate information. As an example, Pyre doesn’t know something about Databricks secret scopes, so we describe the API as being a supply of person credentials. This then permits the instrument to trace these credentials to any potential sinks that ought to be alerted on, resembling a print or logging assertion. We have supplied a set of instance scripts and configurations for Pyre and Pysa as a place to begin, however you need to outline your personal guidelines as wanted.

Beneath, you may see an instance of Pysa taint annotation guidelines we outlined for Databricks utilities:

### dbutils
def dbutils.secrets and techniques.get(scope, key) -> TaintSource[UserSecrets]: ...

def dbutils.secrets and techniques.getBytes(scope, key) -> TaintSource[UserSecrets]: ...

def dbutils.credentials.getCurrentCredentials() -> TaintSource[UserSecrets]: ...

def dbutils.jobs.taskValues.set(key, worth: TaintSink[RequestSend_DATA]): ...

def dbutils.pocket book.run(path, timeout_seconds, arguments: TaintSink[RequestSend_DATA]): ...

def dbutils.fs.mount(supply, mountPoint, encryptionType, extraConfigs: TaintSink[Authentication, DataStorage]): ...

def dbutils.fs.put(file, contents: TaintSink[FileSystem_ReadWrite], overwrite): ...

Some examples of the alerts we enabled are as follows:

Hardcoded Credentials
Customers shouldn’t be utilizing hardcoded, cleartext credentials in code. This contains AWS IAM credentials which are set in Spark properties or different libraries. We do that utilizing a literal string comparability that identifies these values as credentials which get tracked to APIs that take authentication parameters. Utilizing credentials on this method can simply result in leaks in supply management, logs, or simply from sharing entry to notebooks with different unauthorized customers. For those who get alerted to this concern, the credentials ought to be revoked and the code up to date to take away the hardcoded values.

Credential Leaks
If customers have both hardcoded credentials or utilizing secret scopes, they shouldn’t be logging or printing out these values as that might expose them to unauthorized customers. Additionally, credentials shouldn’t be handed as parameters to pocket book workflows as that may trigger them to seem in logs or probably be seen to unauthorized customers. If that is detected then these credentials ought to be revoked and the code up to date to take away the offending code. For pocket book workflows, fairly than passing secrets and techniques you may go a scope title as a parameter to the kid pocket book.

Insecure Configuration
Databricks clusters usually have cluster-scoped credentials, resembling Occasion Profiles or Azure service principal secrets and techniques. With Unity Catalog, we really put off this notion in favor of scoped-down, non permanent, per-user tokens. Nevertheless, if customers are setting credentials programmatically resembling within the SparkSession configuration, international Hadoop configuration, or DBFS mounts, we could wish to alert on that because it may result in these credentials being shared throughout completely different customers. We advocate cluster-scoped credentials or Unity Catalog as an alternative of dynamically setting credentials at runtime.

Reviewing Scan Outcomes

As soon as the scan is accomplished, a report will likely be generated with the outcomes. Within the case of Pysa this can be a JSON file that may be parsed and formatted for assessment. In our instance we offer an embedded report with hyperlinks to the notebooks which can have points to assessment. Pysa studies may also be considered with the Static Evaluation Put up Processor (SAPP) instrument that’s a part of the Pyre/Pysa venture. To make use of SAPP with the output you will have to obtain the JSON output recordsdata from the cluster to your native machine the place you may run SAPP. Whereas the pocket book command logs present us a view of the code run at that time limit, the code or the pocket book itself could have modified or been deleted.

We have supplied a Databricks repo with code and instance Pyre configurations you can begin with. You need to customise the principles and configuration primarily based in your safety necessities.

For extra details about Databricks safety, please go to our Safety & Belief Heart or contact [email protected].