Banner


Workshop 2.4: MSTICPy


- Microsoft Threat Intelligence Center Jupyter & Python Security Tools

Contributors:

  • Ian Hellen (@ianhellen)

  • Pete Bryan (@MSSPete)

  • Ashwin Patil (@ashwinpatil)

Contents


MSTICPy is a library for InfoSec investigation and hunting in Jupyter Notebooks. It includes functionality to:

  • query log data from multiple sources

  • enrich the data with Threat Intelligence, geolocations and Azure resource data

  • extract Indicators of Activity (IoA) from logs and unpack encoded data

  • perform sophisticated analysis such as anomalous session detection and time series decomposition

  • visualize data using interactive timelines, process trees and multi-dimensional Morph Charts

It also includes some time-saving notebook tools such as widgets to set query time boundaries, select and display items from lists, and configure the notebook environment.

Source Code: https://github.com/microsoft/msticpy Python Package: https://pypi.org/project/msticpy/#:~:text=Microsoft Threat Intelligence Python Security Tools. msticpy is,functionality to: query log data from multiple sources Docs: https://msticpy.readthedocs.io/en/latest/


Basics - installing and configuring

To use any library in Python you first need to install the pacakge and import it. There are several ways to do this depending on how you want to access the library, however the simplest and easiest is using pip. Pip is the pacakge installer for Python and makes finding and installing Python pacakges simple. You can use pip to install packages via the command line, or if you are using a notebook, directly in a notebook cell. Azure ML compute come with Pip installed already but if you are running your notebook elsewhere you may need to install pip first.

To do this we need to use %pip followed by install and the pacakge name. e.g.: %pip install msticpy

MSTICPy is a library with a broad range of functionality. As such installing the whole library can be more than required for a lot of uses. As such MSTICPy has implemented a series of Extras that allow for the installation of certain part of the library. These Extras are grouped around core technologies that you might want to use with MSTICPy.

Extra

Functionality

–none–

Most functionality (approx 75%) Kqlmagic Jupyter basic

keyvault

Key Vault and keyring storage of settings secrets

azure

Azure API data retrieval, Azure storage APIs, Sentinel APIs

kql

Kqlmagic Jupyter extended functionality

azsentinel

Combination of core install + “azure”, “keyvault”, “kql”

ml

Timeseries analysis, Event clustering, Outlier analysis

splunk

Splunk data queries

vt3

VirusTotal V3 graph API

riskiq

RiskIQ Illuminate threat intel provider & pivot functions

all

Includes all of above packages

dev

Development tools plus “base”

test

“dev” plus “all”

To install a specific Extra, use the following syntax: %pip install msticpy[extra]

You can also install multiple extras at once: %pip install msticpy[extra1,extra2,...]

%pip install msticpy[all]
Requirement already satisfied: msticpy[all] in e:\src\microsoft\msticpy (1.5.0)
Requirement already satisfied: attrs>=18.2.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (21.2.0)
Requirement already satisfied: azure-common>=1.1.18 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.1.27)
Requirement already satisfied: azure-core>=1.2.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.19.0)
Requirement already satisfied: azure-identity>=1.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.5.0)
Requirement already satisfied: azure-mgmt-subscription>=1.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.0.0)
Requirement already satisfied: bokeh>=1.4.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.3.2)
Requirement already satisfied: cryptography>=3.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (3.4.8)
Requirement already satisfied: deprecated>=1.2.4 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.2.10)
Requirement already satisfied: dnspython<=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.16.0)
Requirement already satisfied: folium>=0.9.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.11.0)
Requirement already satisfied: geoip2>=2.9.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (4.1.0)
Requirement already satisfied: html5lib in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.1)
Requirement already satisfied: idna<3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.10)
Requirement already satisfied: ipwhois>=1.1.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.2.0)
Requirement already satisfied: ipython>=7.1.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (7.27.0)
Requirement already satisfied: ipywidgets>=7.4.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (7.6.4)
Requirement already satisfied: KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.1.114.dev26)
Requirement already satisfied: lxml>=4.6.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (4.6.3)
Requirement already satisfied: matplotlib>=3.0.0 in c:\users\ian\appdata\roaming\python\python37\site-packages (from msticpy[all]) (3.3.3)
Requirement already satisfied: msrest>=0.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.6.21)
Requirement already satisfied: msrestazure>=0.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.6.4)
Requirement already satisfied: networkx>=2.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.6.3)
Requirement already satisfied: numpy>=1.15.4 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.21.2)
Requirement already satisfied: pandas>=1.1.5 in c:\users\ian\appdata\roaming\python\python37\site-packages (from msticpy[all]) (1.2.1)
Requirement already satisfied: python-dateutil>=2.8.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.8.1)
Requirement already satisfied: pytz>=2019.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2020.1)
Requirement already satisfied: pyyaml>=3.13 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (5.4.1)
Requirement already satisfied: requests>=2.21.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.26.0)
Requirement already satisfied: setuptools>=40.6.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (54.1.2)
Requirement already satisfied: tldextract>=2.2.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (3.0.2)
Requirement already satisfied: tqdm>=4.36.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (4.60.0)
Requirement already satisfied: urllib3>=1.23 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.25.10)
Requirement already satisfied: azure-keyvault-secrets>=4.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (4.2.0)
Requirement already satisfied: azure-mgmt-compute>=4.6.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (17.0.0)
Requirement already satisfied: azure-mgmt-core>=1.2.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.3.0)
Requirement already satisfied: azure-mgmt-keyvault>=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (9.1.0)
Requirement already satisfied: azure-mgmt-monitor>=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.0.0)
Requirement already satisfied: azure-mgmt-network>=2.7.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (16.0.0)
Requirement already satisfied: azure-mgmt-resource>=16.1.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (20.0.0)
Requirement already satisfied: azure-mgmt-resourcegraph>=8.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (8.0.0)
Requirement already satisfied: azure-storage-blob>=12.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (12.9.0)
Requirement already satisfied: keyring>=13.2.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (23.2.1)
Requirement already satisfied: moz_sql_parser<=4.11.21016,>=4.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (4.9.21002)
Requirement already satisfied: nest_asyncio>=1.4.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.5.1)
Requirement already satisfied: openpyxl>=3.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (3.0.5)
Requirement already satisfied: passivetotal>=2.5.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.5.3)
Requirement already satisfied: scikit-learn>=0.20.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.23.0)
Requirement already satisfied: scipy>=1.1.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.5.4)
Requirement already satisfied: splunk-sdk>=1.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.6.14)
Requirement already satisfied: statsmodels>=0.11.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.12.0)
Requirement already satisfied: sumologic-sdk>=0.1.11 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.1.11)
Requirement already satisfied: vt-graph-api>=1.0.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.0.1)
Requirement already satisfied: vt-py>=0.6.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.6.1)
Requirement already satisfied: six>=1.11.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from azure-core>=1.2.2->msticpy[all]) (1.16.0)
Requirement already satisfied: msal-extensions~=0.3.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from azure-identity>=1.5.0->msticpy[all]) (0.3.0)
Requirement already satisfied: msal<2.0.0,>=1.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from azure-identity>=1.5.0->msticpy[all]) (1.11.0)
Requirement already satisfied: packaging>=16.8 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from bokeh>=1.4.0->msticpy[all]) (21.0)
Requirement already satisfied: pillow>=7.1.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from bokeh>=1.4.0->msticpy[all]) (8.3.1)
Requirement already satisfied: tornado>=5.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from bokeh>=1.4.0->msticpy[all]) (6.1)
Requirement already satisfied: typing-extensions>=3.7.4 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from bokeh>=1.4.0->msticpy[all]) (3.10.0.2)
Requirement already satisfied: Jinja2>=2.9 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from bokeh>=1.4.0->msticpy[all]) (3.0.1)
Requirement already satisfied: cffi>=1.12 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from cryptography>=3.1->msticpy[all]) (1.14.6)
Requirement already satisfied: pycparser in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from cffi>=1.12->cryptography>=3.1->msticpy[all]) (2.20)
Requirement already satisfied: wrapt<2,>=1.10 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from deprecated>=1.2.4->msticpy[all]) (1.12.1)
Requirement already satisfied: branca>=0.3.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from folium>=0.9.0->msticpy[all]) (0.4.2)
Requirement already satisfied: maxminddb<3.0.0,>=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from geoip2>=2.9.0->msticpy[all]) (2.0.3)
Requirement already satisfied: aiohttp<4.0.0,>=3.6.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from geoip2>=2.9.0->msticpy[all]) (3.6.3)
Requirement already satisfied: async-timeout<4.0,>=3.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from aiohttp<4.0.0,>=3.6.2->geoip2>=2.9.0->msticpy[all]) (3.0.1)
Requirement already satisfied: multidict<5.0,>=4.5 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from aiohttp<4.0.0,>=3.6.2->geoip2>=2.9.0->msticpy[all]) (4.7.6)
Requirement already satisfied: yarl<1.6.0,>=1.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from aiohttp<4.0.0,>=3.6.2->geoip2>=2.9.0->msticpy[all]) (1.5.1)
Requirement already satisfied: chardet<4.0,>=2.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from aiohttp<4.0.0,>=3.6.2->geoip2>=2.9.0->msticpy[all]) (3.0.4)
Requirement already satisfied: traitlets>=4.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (5.1.0)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (3.0.20)
Requirement already satisfied: pickleshare in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (0.7.5)
Requirement already satisfied: pygments in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (2.10.0)
Requirement already satisfied: colorama in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (0.4.4)
Requirement already satisfied: matplotlib-inline in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (0.1.2)
Requirement already satisfied: jedi>=0.16 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (0.18.0)
Requirement already satisfied: decorator in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (5.1.0)
Requirement already satisfied: backcall in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (0.2.0)
Requirement already satisfied: jupyterlab-widgets>=1.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipywidgets>=7.4.2->msticpy[all]) (1.0.0)
Requirement already satisfied: nbformat>=4.2.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipywidgets>=7.4.2->msticpy[all]) (5.1.3)
Requirement already satisfied: ipython-genutils~=0.2.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipywidgets>=7.4.2->msticpy[all]) (0.2.0)
Requirement already satisfied: widgetsnbextension~=3.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipywidgets>=7.4.2->msticpy[all]) (3.5.1)
Requirement already satisfied: ipykernel>=4.5.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipywidgets>=7.4.2->msticpy[all]) (6.4.1)
Requirement already satisfied: argcomplete>=1.12.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (1.12.3)
Requirement already satisfied: jupyter-client<8.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (7.0.1)
Requirement already satisfied: debugpy<2.0,>=1.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (1.4.1)
Requirement already satisfied: importlib-metadata<5 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (4.8.1)
Requirement already satisfied: zipp>=0.5 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from importlib-metadata<5->ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (3.6.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from jedi>=0.16->ipython>=7.1.1->msticpy[all]) (0.8.1)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from Jinja2>=2.9->bokeh>=1.4.0->msticpy[all]) (2.0.1)
Requirement already satisfied: pyzmq>=13 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (22.2.1)
Requirement already satisfied: entrypoints in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (0.3)
Requirement already satisfied: jupyter-core>=4.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (4.8.1)
Requirement already satisfied: pywin32>=1.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from jupyter-core>=4.6.0->jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (228)
Requirement already satisfied: pywin32-ctypes!=0.1.0,!=0.1.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from keyring>=13.2.1->msticpy[all]) (0.2.0)
Requirement already satisfied: prettytable>=0.7.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (0.7.2)
Requirement already satisfied: pyperclip>=1.7.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (1.8.0)
Requirement already satisfied: beautifulsoup4>=4.6.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (4.9.3)
Requirement already satisfied: password-strength>=0.0.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (0.0.3.post2)
Requirement already satisfied: Markdown>=3.0.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (3.3.4)
Requirement already satisfied: flask>=1.0.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (1.1.2)
Requirement already satisfied: psutil>=5.4.7 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (5.7.2)
Requirement already satisfied: isodate>=0.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (0.6.0)Note: you may need to restart the kernel to use updated packages.

Requirement already satisfied: soupsieve>1.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from beautifulsoup4>=4.6.3->KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (2.2.1)
Requirement already satisfied: click>=5.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from flask>=1.0.3->KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (7.1.2)
Requirement already satisfied: Werkzeug>=0.15 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from flask>=1.0.3->KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (1.0.1)
Requirement already satisfied: itsdangerous>=0.24 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from flask>=1.0.3->KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (1.1.0)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from matplotlib>=3.0.0->msticpy[all]) (1.3.1)
Requirement already satisfied: cycler>=0.10 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from matplotlib>=3.0.0->msticpy[all]) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from matplotlib>=3.0.0->msticpy[all]) (2.4.7)
Requirement already satisfied: mo-dots==4.2.20340 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from moz_sql_parser<=4.11.21016,>=4.5.0->msticpy[all]) (4.2.20340)
Requirement already satisfied: mo-future==3.147.20327 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from moz_sql_parser<=4.11.21016,>=4.5.0->msticpy[all]) (3.147.20327)
Requirement already satisfied: mo-logs==4.3.20340 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from moz_sql_parser<=4.11.21016,>=4.5.0->msticpy[all]) (4.3.20340)
Requirement already satisfied: mo-imports==3.149.20327 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from mo-dots==4.2.20340->moz_sql_parser<=4.11.21016,>=4.5.0->msticpy[all]) (3.149.20327)
Requirement already satisfied: mo-kwargs==3.93.20259 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from mo-logs==4.3.20340->moz_sql_parser<=4.11.21016,>=4.5.0->msticpy[all]) (3.93.20259)
Requirement already satisfied: PyJWT[crypto]<3,>=1.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msal<2.0.0,>=1.6.0->azure-identity>=1.5.0->msticpy[all]) (1.7.1)
Requirement already satisfied: portalocker~=1.6 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msal-extensions~=0.3.0->azure-identity>=1.5.0->msticpy[all]) (1.7.1)
Requirement already satisfied: requests-oauthlib>=0.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msrest>=0.6.0->msticpy[all]) (1.3.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msrest>=0.6.0->msticpy[all]) (2021.10.8)
Requirement already satisfied: adal<2.0.0,>=0.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msrestazure>=0.6.0->msticpy[all]) (1.2.4)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbformat>=4.2.0->ipywidgets>=7.4.2->msticpy[all]) (3.2.0)
Requirement already satisfied: pyrsistent>=0.14.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.4.2->msticpy[all]) (0.17.3)
Requirement already satisfied: et-xmlfile in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from openpyxl>=3.0->msticpy[all]) (1.0.1)
Requirement already satisfied: jdcal in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from openpyxl>=3.0->msticpy[all]) (1.4.1)
Requirement already satisfied: future in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from passivetotal>=2.5.3->msticpy[all]) (0.18.2)
Requirement already satisfied: ez-setup in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from passivetotal>=2.5.3->msticpy[all]) (0.9)
Requirement already satisfied: wcwidth in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=7.1.1->msticpy[all]) (0.2.5)
Requirement already satisfied: charset-normalizer~=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from requests>=2.21.1->msticpy[all]) (2.0.4)
Requirement already satisfied: oauthlib>=3.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from requests-oauthlib>=0.5.0->msrest>=0.6.0->msticpy[all]) (3.0.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from scikit-learn>=0.20.2->msticpy[all]) (2.1.0)
Requirement already satisfied: joblib>=0.11 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from scikit-learn>=0.20.2->msticpy[all]) (1.0.1)
Requirement already satisfied: patsy>=0.5 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from statsmodels>=0.11.1->msticpy[all]) (0.5.1)
Requirement already satisfied: requests-file>=1.4 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from tldextract>=2.2.2->msticpy[all]) (1.5.1)
Requirement already satisfied: filelock>=3.0.8 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from tldextract>=2.2.2->msticpy[all]) (3.0.12)
Requirement already satisfied: futures in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from vt-graph-api>=1.0.1->msticpy[all]) (3.1.1)
Requirement already satisfied: notebook>=4.4.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (6.4.3)
Requirement already satisfied: Send2Trash>=1.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (1.8.0)
Requirement already satisfied: terminado>=0.8.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.9.4)
Requirement already satisfied: argon2-cffi in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (20.1.0)
Requirement already satisfied: prometheus-client in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.11.0)
Requirement already satisfied: nbconvert in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (6.0.7)
Requirement already satisfied: pywinpty>=0.5 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from terminado>=0.8.3->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.5.7)
Requirement already satisfied: webencodings in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from html5lib->msticpy[all]) (0.5.1)
Requirement already satisfied: defusedxml in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.7.1)
Requirement already satisfied: jupyterlab-pygments in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.1.2)
Requirement already satisfied: testpath in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.5.0)
Requirement already satisfied: bleach in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (4.0.0)
Requirement already satisfied: pandocfilters>=1.4.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (1.4.3)
Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.5.3)
Requirement already satisfied: mistune<2,>=0.8.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.8.4)
Requirement already satisfied: async-generator in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (1.10)

Configuration

Once installed MSTICPy can be imported in the same way as any other Python library, however to make things a bit easier we have created the init_notebook function that will automatically import the library and configure it for use in a Jupyter Notebook.

Note: Passing `globals()` lets the init function import stuff into the notebook top-level namespace

import msticpy
msticpy.init_notebook(globals())

Starting Notebook initialization...

msticpy version installed: 1.5.0rc3 latest published: 1.5.0
A newer version of msticpy - 1.5.0 is available.
Upgrade with pip install --upgrade msticpy

Processing imports....
Imported: pd (pandas), IPython.get_ipython, IPython.display.display, IPython.display.HTML, IPython.display.Markdown, widgets (ipywidgets), pathlib.Path, plt (matplotlib.pyplot), matplotlib.MatplotlibDeprecationWarning, sns (seaborn), np (numpy), msticpy, msticpy.data.QueryProvider, msticpy.nbtools.foliummap.FoliumMap, msticpy.common.utility.md, msticpy.common.utility.md_warn, msticpy.common.wsconfig.WorkspaceConfig, msticpy.datamodel.pivot.Pivot, msticpy.datamodel.entities, msticpy.vis.mp_pandas_plot
Checking configuration....
No valid configuration for Azure Sentinel found.
Azure CLI not detected. AzCLI single sign-on disabled ({_CLI_WIKI_MSSG_SHORT})
Setting notebook options....

Notebook setup completed with some warnings.

One or more configuration items were missing or set incorrectly.

Please run the Getting Started Guide for Azure Sentinel ML Notebooks notebook. and the msticpy configuration guide.

This notebook may still run but with reduced functionality.

Notebook initialization complete

False

MSTICPy’s config file

MSTICPy can handle connections to a variety of data sources and services, including Azure Sentinel.

To make it easier to manage and re-use the configuration and credentials fo these things MSTICPy has its own config file that holds these items - msticpyconfig.yaml

If you didn’t have a msticpyconfig.yaml file in your workspace folder (which is likely if this is your first use of MSTICPY), the init_notebook function should have created one for you and populated it file structure you are in.

You can populate msticpyconfig manually or you can used MSTICPy’s settings editor to view and edit the settings stored there.

msticpy.MpConfigEdit()

There are different settings depending on what feature of MSTICPy you are setting configuration for. This includes config settings for connecting to Security data sources such as Microsoft Sentinel, Threat Intelligence providers such as VirusTotal, or for connecting to KeyVault if you want to use that service for securely storing your MSTICPy settings.

Once you have completed a these sections you can check that you have valid settings by:

  • using the Validate Settings button

  • using the msticpy.settings.settings attribute to display current settings.

Tip: If you edit the settings, call msticpy.settings.refresh_config() to reload the settings.

import yaml
print(yaml.safe_dump(msticpy.settings.settings)[:500])
DataProviders: {}
QueryDefinitions:
  Default:
  - queries
msticpy:
  FriendlyExceptions: true

More details on populating the config file can be found in the MSTICPy Settings Editor Documentation


MSTICPy Widgets

MSTICPy include a series of Notebook widgets to make interacting with data easier, especially for users without a programming background.

The widgets are designed to fulfil a number of common tasks that a user might need to interact with a notebook such as select items from returned data, or set a timeframe for a query.

The widgets themselves are build in ipywidgets and are available in the msticpy.nbtools.nbwidgets module.

Note: Widgets are automatically imported by init_notebook

The below code creates our Time Range widget that can be used to allow a user to set a time range. We are telling it to use days as its unit of measurements and set a max range to select from.

from msticpy.nbtools.nbwidgets import *

time_select = QueryTime(units="day", max_before=20, before=5, max_after=1)
time_select.display()

We can then call the start end end properties and get datetime objects based on the user selection.

time_select.start
datetime.datetime(2021, 11, 26, 18, 22, 2, 56675)

Other widgets allow for the selection of items from list, along with a text filter option to help users find items:

items = ["item 1", "item 2", "item 3"]
selection = SelectItem(item_list=items, description="Select item", auto_display=True)

There are also security specific widgets such as SelectAlert which allows a user to select a specific alert from a list of alerts. With this widget and others you can also specify an action, this is a follow on funciton that is executed with the value of the value selected in the widget.

In the cell below we are using the action method to display the selected alert.

import pandas as pd
from msticpy.nbtools.nbdisplay import display_alert

# As discussed earlier pands read_* functions can call remote files as well as local ones.
alerts = pd.read_pickle("https://github.com/microsoft/msticpy/raw/main/tests/testdata/localdata/alerts_list.pkl")
alert_select = SelectAlert(alerts=alerts, action=display_alert)
alert_select.display()

Selected Alert: 'SSH Anomalous Login ML'

Alert_time: 2019-02-18 01:49:02,  Compr_entity:Alert_id: f1ce87ca-8863-4a66-a0bd-a4d3776a7c64
0
TenantId 52b1ab41-869e-4138-9e40-2a4457f09bf0
TimeGenerated 2019-02-18 02:29:07
AlertDisplayName SSH Anomalous Login ML
AlertName SSH Anomalous Login ML
Severity Low
Description Anomalous login detected for SSH account
ProviderName CustomAlertRule
VendorName Alert Rule
VendorOriginalId b0e143b8-4fa8-47bc-8bc1-9780c8b75541
SystemAlertId f1ce87ca-8863-4a66-a0bd-a4d3776a7c64
ResourceId
SourceComputerId
AlertType CustomAlertRule_0a4e5f7c-9756-45f8-83c4-94c756844698
ConfidenceLevel Unknown
ConfidenceScore NaN
IsIncident False
StartTimeUtc 2019-02-18 01:49:02
EndTimeUtc 2019-02-18 02:19:02
ProcessingEndTime 2019-02-18 02:29:07
RemediationSteps
ExtendedProperties {'Alert Mode': 'Aggregated', 'Search Query': '{"detailBladeInputs":{"id":"/subscriptions/40dcc8b...
Entities [{'$id': '3', 'Address': '23.97.60.214', 'Type': 'ip', 'Count': 1}, {'$id': '4', 'HostName': 'MS...
SourceSystem Detection
WorkspaceSubscriptionId 40dcc8bf-0478-4f3b-b275-ed0a94f2c013
WorkspaceResourceGroup asihuntomsworkspacerg
ExtendedLinks
ProductName
ProductComponentName
AlertLink
Type SecurityAlert
CompromisedEntity

Some widgets are registered, meaning that they can be re-used later and will rememeber previous values entered.
This can be done by simply creating a widget with the same parameters as previously.

mem_text = GetText(prompt="Enter your name")
mem_text
mem_text2 = GetText(prompt="Enter your name")
mem_text2

Other MSTICPy widgets include:

  • A simple datetime based lookback slider Lookback

  • A text box to capture user input GetText

  • A widget to capture and return an Environemnt Variable GetEnvrionmentKey

  • A widget to select a subset of items from a list SelectSubset

  • A widget to show progress of a task Progress

  • Multi option buttons with a wait function that pauses cell execution until a user selects an option OptionButtons

More details on MSTICPy’s widgets can be found here: https://msticpy.readthedocs.io/en/latest/visualization/NotebookWidgets.html


Query providers revisited [Ian]

Supported providers

  • Microsoft Sentinel

  • Microsoft Defender/Defender for Endpoint

  • Splunk

  • Sumologic

  • Microsoft Graph

  • Local data

  • Mordor/Security Datasets

  • Kusto/Azure Data Explorer

  • Azure Resource Graph

from msticpy.data import QueryProvider
import pandas as pd

# Load query providers (typically you'll be using just one)
qry_prov_az = QueryProvider("AzureSentinel")
qry_prov_sp = QueryProvider("Splunk")
qry_prov_mde = QueryProvider("MDE")

# Special provider that uses local data files
qry_prov_loc = QueryProvider("LocalData", data_paths=["./data"], query_paths=["./data"])
Please wait. Loading Kqlmagic extension...done
Warning: Custom query definitions path data not found

list_queries and running a query

qry_prov_az.list_queries()[:10]
['Azure.get_vmcomputer_for_host',
 'Azure.get_vmcomputer_for_ip',
 'Azure.list_aad_signins_for_account',
 'Azure.list_aad_signins_for_ip',
 'Azure.list_all_signins_geo',
 'Azure.list_azure_activity_for_account',
 'Azure.list_azure_activity_for_ip',
 'Azure.list_azure_activity_for_resource',
 'Azure.list_storage_ops_for_hash',
 'Azure.list_storage_ops_for_ip']
qry_prov_az.Azure.list_aad_signins_for_account?
Signature:       qry_prov_az.Azure.list_aad_signins_for_account(*args, **kwargs) -> Union[pandas.core.frame.DataFrame, Any]
Call signature:  qry_prov_az.Azure.list_aad_signins_for_account(*args, **kwargs)
Type:            partial
String form:     functools.partial(<bound method QueryProvider._execute_query of <msticpy.data.data_providers.Quer <...> er object at 0x000001F0BCA131C8>>, query_path='Azure', query_name='list_aad_signins_for_account')
File:            c:\users\ian\anaconda3\envs\condadev\lib\functools.py
Docstring:      
Lists Azure AD Signins for Account

Parameters
----------
account_name: str
    The account name to find
add_query_items: str (optional)
    Additional query clauses
end: datetime (optional)
    Query end time
start: datetime (optional)
    Query start time
    (default value is: -5)
table: str (optional)
    Table name
    (default value is: SigninLogs)
Class docstring:
partial(func, *args, **keywords) - new function with partial application
of the given arguments and keywords.

Query time ranges

Each provider has a built-in (modifiable) time range.

This supplies time parameters:

  • If the query requires them

  • You have not overridden them

  • The query template does not provide its own default values

qry_prov_az.query_time

Query parameters

qry_prov_az.browse()

Lists Azure Activity for Caller IP Address(es)

Parameters

add_query_items: str (optional)
Additional query clauses
end: datetime (optional)
Query end time
ip_address_list: list
The IP Address or list of Addresses
start: datetime (optional)
Query start time
(default value is: -5)
table: str (optional)
Table name
(default value is: AzureActivity)

Query

{table} 
| where CallerIpAddress in ({ip_address_list}) 
| where TimeGenerated >= datetime({start}) 
| where TimeGenerated <= datetime({end}) {add_query_items}

Example

{QueryProvider}[.QueryPath].QueryName(params...)

qry_prov.Azure.list_azure_activity_for_ip(start=start, end=end, hostname=host)
{table}
| where CallerIpAddress in ({ip_address_list})
| where TimeGenerated >= datetime({start})
| where TimeGenerated <= datetime({end})
{add_query_items}

qry_prov_az.connect(WorkspaceConfig("CyberSecuritySOC"))
qry_prov_az.Azure.list_azure_activity_for_ip()
OperationName OperationNameValue Level ActivityStatus ActivityStatusValue ActivitySubstatus ActivitySubstatusValue ResourceGroup SubscriptionId CorrelationId Caller CallerIpAddress Category CategoryValue HTTPRequest Properties EventSubmissionTimestamp Authorization ResourceId OperationId ResourceProvider ResourceProviderValue Resource EventDataId TenantId TimeGenerated SourceSystem Authorization_d Claims Claims_d Properties_d Hierarchy Type _ResourceId
qry_prov_az.Azure.list_azure_activity_for_ip(
    ip_address_list="176.199.184.128", add_query_items="| take 3"
)
OperationName OperationNameValue Level ActivityStatus ActivityStatusValue ActivitySubstatus ActivitySubstatusValue ResourceGroup SubscriptionId CorrelationId Caller CallerIpAddress Category CategoryValue HTTPRequest Properties EventSubmissionTimestamp Authorization ResourceId OperationId ResourceProvider ResourceProviderValue Resource EventDataId TenantId TimeGenerated SourceSystem Authorization_d Claims Claims_d Properties_d Hierarchy Type _ResourceId
0 Check user authorization and license Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action Informational Started Started soc d1d8779d-38d7-4f06-91db-9cbc8de0176f 2aaa092a-7b3c-4a77-a056-57ae1a59f7aa Markus.Falkowski@microsoft.com 176.199.184.128 Administrative Administrative {\r\n "clientRequestId": "8bdab8e4-50bb-4108-a814-1baa55ebe019",\r\n "clientIpAddress": "176.1... {\r\n "eventCategory": "Administrative",\r\n "entity": "/subscriptions/d1d8779d-38d7-4f06-91db... 2021-11-29 11:22:17.142000+00:00 {\r\n "action": "Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action",\r\n "scop... /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourceGroups/soc/providers/Microsoft.Opera... 2ddbaaaa-246a-4fcc-aa54-3081fee17dbc Microsoft.SecurityInsights Microsoft.SecurityInsights Microsoft.SecurityInsights c682a3e7-7162-4a65-bdbf-98b54a6374b1 8ecf8077-cf51-4820-aadd-14040956f35d 2021-11-29 11:20:14.994000+00:00 Azure None None None AzureActivity /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourcegroups/soc/providers/microsoft.opera...
1 Check user authorization and license Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action Informational Started Started soc d1d8779d-38d7-4f06-91db-9cbc8de0176f 2aaa092a-7b3c-4a77-a056-57ae1a59f7aa Markus.Falkowski@microsoft.com 176.199.184.128 Administrative Administrative {\r\n "clientRequestId": "8bdab8e4-50bb-4108-a814-1baa55ebe019",\r\n "clientIpAddress": "176.1... {\r\n "eventCategory": "Administrative",\r\n "entity": "/subscriptions/d1d8779d-38d7-4f06-91db... 2021-11-29 11:21:33.173000+00:00 {\r\n "action": "Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action",\r\n "scop... /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourceGroups/soc/providers/Microsoft.Opera... 945c61d7-e243-4ced-b5b8-66e1e93f7369 Microsoft.SecurityInsights Microsoft.SecurityInsights Microsoft.SecurityInsights 529832af-de9a-40d9-b4d1-a82a5da34302 8ecf8077-cf51-4820-aadd-14040956f35d 2021-11-29 11:20:14.995000+00:00 Azure None None None AzureActivity /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourcegroups/soc/providers/microsoft.opera...
2 Check user authorization and license Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action Informational Started Started soc d1d8779d-38d7-4f06-91db-9cbc8de0176f 2aaa092a-7b3c-4a77-a056-57ae1a59f7aa Markus.Falkowski@microsoft.com 176.199.184.128 Administrative Administrative {\r\n "clientRequestId": "8bdab8e4-50bb-4108-a814-1baa55ebe019",\r\n "clientIpAddress": "176.1... {\r\n "eventCategory": "Administrative",\r\n "entity": "/subscriptions/d1d8779d-38d7-4f06-91db... 2021-11-29 11:21:39.153000+00:00 {\r\n "action": "Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action",\r\n "scop... /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourceGroups/soc/providers/Microsoft.Opera... 9cb7dbe5-e3f7-4dad-95e8-b46da3d4ac72 Microsoft.SecurityInsights Microsoft.SecurityInsights Microsoft.SecurityInsights fcbe3a89-a8d2-4796-aa38-70dfa32461f8 8ecf8077-cf51-4820-aadd-14040956f35d 2021-11-29 11:20:15.001000+00:00 Azure None None None AzureActivity /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourcegroups/soc/providers/microsoft.opera...

DateTime parameters are special

Can supply:

  • Python datetimes

  • Date string

  • integers

    • 0 = now

    • negative numbers == # of days (or partial days) before now

    • positive numbers == # of days (or partial days) after now

from datetime import datetime
str(datetime.utcnow())
'2021-11-30 03:15:57.033469'
from datetime import datetime

qry_prov_az.Azure.list_azure_activity_for_ip(
    ip_address_list="176.199.184.128",
    start=-1.5,
    end=str(datetime.utcnow()),
    add_query_items="| take 3"
)
OperationName OperationNameValue Level ActivityStatus ActivityStatusValue ActivitySubstatus ActivitySubstatusValue ResourceGroup SubscriptionId CorrelationId Caller CallerIpAddress Category CategoryValue HTTPRequest Properties EventSubmissionTimestamp Authorization ResourceId OperationId ResourceProvider ResourceProviderValue Resource EventDataId TenantId TimeGenerated SourceSystem Authorization_d Claims Claims_d Properties_d Hierarchy Type _ResourceId
0 Check user authorization and license Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action Informational Started Started soc d1d8779d-38d7-4f06-91db-9cbc8de0176f 2aaa092a-7b3c-4a77-a056-57ae1a59f7aa Markus.Falkowski@microsoft.com 176.199.184.128 Administrative Administrative {\r\n "clientRequestId": "8bdab8e4-50bb-4108-a814-1baa55ebe019",\r\n "clientIpAddress": "176.1... {\r\n "eventCategory": "Administrative",\r\n "entity": "/subscriptions/d1d8779d-38d7-4f06-91db... 2021-11-29 11:22:17.142000+00:00 {\r\n "action": "Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action",\r\n "scop... /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourceGroups/soc/providers/Microsoft.Opera... 2ddbaaaa-246a-4fcc-aa54-3081fee17dbc Microsoft.SecurityInsights Microsoft.SecurityInsights Microsoft.SecurityInsights c682a3e7-7162-4a65-bdbf-98b54a6374b1 8ecf8077-cf51-4820-aadd-14040956f35d 2021-11-29 11:20:14.994000+00:00 Azure None None None AzureActivity /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourcegroups/soc/providers/microsoft.opera...
1 Check user authorization and license Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action Informational Started Started soc d1d8779d-38d7-4f06-91db-9cbc8de0176f 2aaa092a-7b3c-4a77-a056-57ae1a59f7aa Markus.Falkowski@microsoft.com 176.199.184.128 Administrative Administrative {\r\n "clientRequestId": "8bdab8e4-50bb-4108-a814-1baa55ebe019",\r\n "clientIpAddress": "176.1... {\r\n "eventCategory": "Administrative",\r\n "entity": "/subscriptions/d1d8779d-38d7-4f06-91db... 2021-11-29 11:21:33.173000+00:00 {\r\n "action": "Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action",\r\n "scop... /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourceGroups/soc/providers/Microsoft.Opera... 945c61d7-e243-4ced-b5b8-66e1e93f7369 Microsoft.SecurityInsights Microsoft.SecurityInsights Microsoft.SecurityInsights 529832af-de9a-40d9-b4d1-a82a5da34302 8ecf8077-cf51-4820-aadd-14040956f35d 2021-11-29 11:20:14.995000+00:00 Azure None None None AzureActivity /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourcegroups/soc/providers/microsoft.opera...
2 Check user authorization and license Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action Informational Started Started soc d1d8779d-38d7-4f06-91db-9cbc8de0176f 2aaa092a-7b3c-4a77-a056-57ae1a59f7aa Markus.Falkowski@microsoft.com 176.199.184.128 Administrative Administrative {\r\n "clientRequestId": "8bdab8e4-50bb-4108-a814-1baa55ebe019",\r\n "clientIpAddress": "176.1... {\r\n "eventCategory": "Administrative",\r\n "entity": "/subscriptions/d1d8779d-38d7-4f06-91db... 2021-11-29 11:21:39.153000+00:00 {\r\n "action": "Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action",\r\n "scop... /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourceGroups/soc/providers/Microsoft.Opera... 9cb7dbe5-e3f7-4dad-95e8-b46da3d4ac72 Microsoft.SecurityInsights Microsoft.SecurityInsights Microsoft.SecurityInsights fcbe3a89-a8d2-4796-aa38-70dfa32461f8 8ecf8077-cf51-4820-aadd-14040956f35d 2021-11-29 11:20:15.001000+00:00 Azure None None None AzureActivity /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourcegroups/soc/providers/microsoft.opera...

add_query_items parameter

Add arbitrary query statements

qry_prov_az.Azure.list_azure_activity_for_ip(
    ip_address_list="176.199.184.128",
    add_query_items="| summarize count() by OperationName"
)
OperationName count_
0 360
1 Check user authorization and license 181
2 Microsoft.SecurityInsights/Incidents/investigations/write 12
3 Gets workflow recommend operation groups 46

Splitting queries

Divide queries into equal time chunks. Useful for queries that exceed timeouts or data limits

Caveats:

  • Only works with template queries

  • Won’t work correctly with joins in some cases

  • Won’t work correctly with aggregates/summaries

  • You must supply start and end parameters explicitly (datetime values)

from datetime import timedelta
df = qry_prov_az.Azure.list_azure_activity_for_ip(
    ip_address_list="176.199.184.128",
    start=datetime.now() - timedelta(days=1),
    end=datetime.utcnow(),
    split_query_by="6H",
)

df[["OperationName", "TimeGenerated"]].groupby("OperationName").count()
Running: 100%|██████████| 5/5 [00:09<00:00,  1.84s/sub-queries]
TimeGenerated
OperationName
276
Check user authorization and license 140
Gets workflow recommend operation groups 46
Microsoft.SecurityInsights/Incidents/investigations/write 8

Creating Custom queries - Basic format

YAML files

Kusto/Log Analytics example

metadata:
  version: 1
  description: Linux Syslog Host Activity Queries
  data_environments: [LogAnalytics]
  data_families: [LinuxSyslog]
  tags: ['linux', 'syslog']
defaults:
  metadata:
    data_source: 'linux_syslog'
    pivot:
      direct_func_entities:
        - Host
  parameters:
      table:
        description: Table name
        type: str
        default: 'Syslog'
      start:
        description: Query start time
        type: datetime
      end:
        description: Query end time
        type: datetime
      add_query_items:
        description: Additional query clauses
        type: str
        default: ''
sources:
  all_syslog:
    description: Returns all syslog activity for a host
    args:
      query: '
         {table}
        | where TimeGenerated >= datetime({start})
        | where TimeGenerated <= datetime({end})
        | where Computer has "{host_name}"
        {add_query_items}'
    parameters:
      host_name:
        description: Host name or FQDN
        type: str
  user_group_activity:
    description: All user/group additions, deletions, and modifications
    args:
      ...

Splunk example

list_all_alerts:
    description: Retrieves all configured alerts
    metadata:
      data_families: [Alerts]
    args:
      query: '
      | rest/servicesNS/-/search/saved/searches
      | search alert.track=1
      | fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule'
    parameters:

OData Example

list_alerts_for_user:
    description: Retrieves list of alerts for a user account
    metadata:
      data_source: 'graph_alert'
    args:
      query: '{path}?$filter=createdDateTime ge {start}
        and createdDateTime le {end}
        and (userStates/any(d:tolower(d/userPrincipalName) eq tolower("{user_principal_name}")
        or userStates/any(d:tolower(d/accountName) eq tolower("{account_name}"))
        {add_query_items}'
      uri: None

Specifying custom path for queries

At runtime or in config

qry_prov = QueryProvider("M365", query_paths=["./myqueries"])

msticpyconfig.yaml

QueryDefinitions:
  Custom:
    - ~/my_queries
    - ~/my_other_queries

Enrichment - Threat intelligence

Note: These sections will use a number of live services, you may not have credentials for these services so they may not run correctly for you.

A very common element of security investigations is to enrich data with Threat Intelligence data from external parties. To support his MSTICPy has integrations with a number of Threat Intelligence providers including:

  • VirusTotal

  • IBM XForce

  • RiskIQ

  • AlienVault OTX

  • GreyNoise

Note: Many of these integrations require an authentication key to be able to use. You can specify these in your msticpyconfig file.

Threat Intelligence integrations can be accessed using the TILookup class or via pivot providers. In the demo below we will use both methods.

# First we create our provider
ti_lookup = TILookup()
# Then we lookup results
ti_results = ti_lookup.lookup_ioc("91.211.89.33")
# Convert results to a DataFrame for ease of viewing
ti_results = ti_lookup.result_to_df(ti_results)
ti_results
Using Open PageRank. See https://www.domcop.com/openpagerank/what-is-openpagerank
Ioc IocType QuerySubtype Provider Result Severity Details RawResult Reference Status
OTX 91.211.89.33 ipv4 None OTX True high {'pulse_count': 3, 'names': ['Public report on attacks in Middle East we attribute to WIRTE APT'... {'whois': 'http://whois.domaintools.com/91.211.89.33', 'reputation': 0, 'indicator': '91.211.89.... https://otx.alienvault.com/api/v1/indicators/IPv4/91.211.89.33/general 0
OPR 91.211.89.33 ipv4 None OPR False information IoC type ipv4 not supported. None None 1
RiskIQ 91.211.89.33 ipv4 None RiskIQ False unknown ERROR: Error #402 "Reputation score not included in license level" (https://api.passivetotal.org... Error #402 "Reputation score not included in license level" (https://api.passivetotal.org/v2/rep... https://community.riskiq.com 3
Tor 91.211.89.33 ipv4 None Tor True information Not found. None https://check.torproject.org/exit-addresses 0
VirusTotal 91.211.89.33 ipv4 None VirusTotal True information {'verbose_msg': 'IP address in dataset', 'response_code': 1, 'positives': 0, 'detected_urls': []} {'asn': 206638, 'undetected_urls': [['http://91.211.89.33/', '534745024fb0a0b687cf79d0f037042a07... https://www.virustotal.com/vtapi/v2/ip-address/report 0
XForce 91.211.89.33 ipv4 None XForce True information {'score': 1, 'cats': {}, 'categoryDescriptions': {}, 'reason': 'Regional Internet Registry', 're... {'ip': '91.211.89.33', 'history': [{'created': '2012-03-22T07:26:00.000Z', 'reason': 'Regional I... https://api.xforce.ibmcloud.com/ipr/91.211.89.33 0
# We can also display them in a browser
TILookup.browse_results(ti_results)

91.211.89.33

Type: 'ipv4', Provider: OTX, severity: high

Details

OTX
pulse_count3
names['Public report on attacks in Middle East we attribute to WIRTE APT', 'WIRTE’s campaign in the Middle East ‘living off the land’ since at least 2019', 'test2']
tags[['wirte', 'macros', 'microsoft excel', 'spear phishing'], ['hkcu', 'class ids', 'appdata', 'programdata'], []]
references[['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'], ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'], []]

Reference:

https://otx.alienvault.com/api/v1/indicators/IPv4/91.211.89.33/general

Raw Results

Raw results from provider...
{'accuracy_radius': 500,
 'area_code': 0,
 'asn': 'AS206638 PE Brezhnev Daniil',
 'base_indicator': {'access_reason': '',
                    'access_type': 'public',
                    'content': '',
                    'description': '',
                    'id': 2822432629,
                    'indicator': '91.211.89.33',
                    'title': '',
                    'type': 'IPv4'},
 'charset': 0,
 'city': 'Dnipro',
 'city_data': True,
 'continent_code': 'EU',
 'country_code': 'UA',
 'country_code2': 'UA',
 'country_code3': 'UKR',
 'country_name': 'Ukraine',
 'dma_code': 0,
 'false_positive': [],
 'flag_title': 'Ukraine',
 'flag_url': '/assets/images/flags/ua.png',
 'indicator': '91.211.89.33',
 'latitude': 48.4735,
 'longitude': 35.046,
 'postal_code': '49028',
 'pulse_info': {'count': 3,
                'pulses': [{'TLP': 'white',
                            'adversary': 'WIRTE',
                            'attack_ids': [{'display_name': 'T1059 - Command '
                                                            'and Scripting '
                                                            'Interpreter',
                                            'id': 'T1059',
                                            'name': 'Command and Scripting '
                                                    'Interpreter'},
                                           {'display_name': 'T1218 - Signed '
                                                            'Binary Proxy '
                                                            'Execution',
                                            'id': 'T1218',
                                            'name': 'Signed Binary Proxy '
                                                    'Execution'},
                                           {'display_name': 'T1113 - Screen '
                                                            'Capture',
                                            'id': 'T1113',
                                            'name': 'Screen Capture'},
                                           {'display_name': 'T1546 - Event '
                                                            'Triggered '
                                                            'Execution',
                                            'id': 'T1546',
                                            'name': 'Event Triggered '
                                                    'Execution'},
                                           {'display_name': 'T1562 - Impair '
                                                            'Defenses',
                                            'id': 'T1562',
                                            'name': 'Impair Defenses'},
                                           {'display_name': 'T1036 - '
                                                            'Masquerading',
                                            'id': 'T1036',
                                            'name': 'Masquerading'},
                                           {'display_name': 'T1566 - Phishing',
                                            'id': 'T1566',
                                            'name': 'Phishing'},
                                           {'display_name': 'T1137.001 - '
                                                            'Office Template '
                                                            'Macros',
                                            'id': 'T1137.001',
                                            'name': 'Office Template Macros'},
                                           {'display_name': 'T1059.001 - '
                                                            'PowerShell',
                                            'id': 'T1059.001',
                                            'name': 'PowerShell'},
                                           {'display_name': 'T1193 - '
                                                            'Spearphishing '
                                                            'Attachment',
                                            'id': 'T1193',
                                            'name': 'Spearphishing '
                                                    'Attachment'}],
                            'author': {'avatar_url': '/otxapi/users/avatar_image/media/avatars/user_2/resized/80/avatar_dacfad0ca8.png',
                                       'id': '2',
                                       'is_following': False,
                                       'is_subscribed': True,
                                       'username': 'AlienVault'},
                            'cloned_from': None,
                            'comment_count': 0,
                            'created': '2021-11-29T16:10:35.042000',
                            'description': 'This February researchers came '
                                           'across MS Excel droppers that use '
                                           'hidden spreadsheets and VBA macros '
                                           'to drop their first stage implant. '
                                           'The implant itself is a VBS script '
                                           'with functionality to collect '
                                           'system information and execute '
                                           'arbitrary code sent by the '
                                           'attackers on the infected machine. '
                                           'Although these intrusion sets may '
                                           'appear similar to the new '
                                           'MuddyWater first stage VBS implant '
                                           'used for reconnaissance and '
                                           'profiling activities, they have '
                                           'slightly different TTPs and wider '
                                           'targeting. To date, most of the '
                                           'known victims are located in the '
                                           'Middle East, but there are also '
                                           'targets in other regions',
                            'downvotes_count': 0,
                            'export_count': 27,
                            'follower_count': 0,
                            'groups': [],
                            'id': '61a4fb7c9b88f16b103c151d',
                            'in_group': False,
                            'indicator_count': 26,
                            'indicator_type_counts': {'FileHash-MD5': 6,
                                                      'FileHash-SHA1': 2,
                                                      'FileHash-SHA256': 2,
                                                      'IPv4': 4,
                                                      'domain': 12},
                            'industries': ['Telecommunications',
                                           'Energy',
                                           'Political',
                                           'Technology',
                                           'Military',
                                           'Financial',
                                           'Diplomatic',
                                           'Government'],
                            'is_author': False,
                            'is_modified': False,
                            'is_subscribing': None,
                            'locked': False,
                            'malware_families': [],
                            'modified': '2021-11-29T16:10:35.042000',
                            'modified_text': '10 hours ago ',
                            'name': 'Public report on attacks in Middle East '
                                    'we attribute to WIRTE APT',
                            'public': 1,
                            'pulse_source': 'web',
                            'references': ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'],
                            'related_indicator_is_active': 1,
                            'related_indicator_type': 'IPv4',
                            'subscriber_count': 166586,
                            'tags': ['wirte',
                                     'macros',
                                     'microsoft excel',
                                     'spear phishing'],
                            'targeted_countries': [],
                            'threat_hunter_has_agents': 1,
                            'threat_hunter_scannable': True,
                            'upvotes_count': 0,
                            'validator_count': 0,
                            'vote': 0,
                            'votes_count': 0},
                           {'TLP': 'white',
                            'adversary': '',
                            'attack_ids': [],
                            'author': {'avatar_url': '/otxapi/users/avatar_image/media/avatars/user_94093/resized/80/avatar_281f69b768.png',
                                       'id': '94093',
                                       'is_following': False,
                                       'is_subscribed': False,
                                       'username': 'Sand-Storm'},
                            'cloned_from': None,
                            'comment_count': 0,
                            'created': '2021-11-29T17:27:20.561000',
                            'description': 'Malicious documents and droppers, '
                                           'as reported by the BBC, have been '
                                           'uncovered by researchers at the '
                                           'University of Kansas in the United '
                                           'States and are subject to an '
                                           'investigation by security firm '
                                           'Kaspersky.',
                            'downvotes_count': 0,
                            'export_count': 2,
                            'follower_count': 0,
                            'groups': [],
                            'id': '61a50d7881106990abd40bd1',
                            'in_group': False,
                            'indicator_count': 23,
                            'indicator_type_counts': {'FileHash-MD5': 6,
                                                      'FileHash-SHA1': 2,
                                                      'FileHash-SHA256': 2,
                                                      'IPv4': 4,
                                                      'domain': 9},
                            'industries': [],
                            'is_author': False,
                            'is_modified': False,
                            'is_subscribing': None,
                            'locked': False,
                            'malware_families': [],
                            'modified': '2021-11-29T17:27:20.561000',
                            'modified_text': '9 hours ago ',
                            'name': 'WIRTE’s campaign in the Middle East '
                                    '‘living off the land’ since at least 2019',
                            'public': 1,
                            'pulse_source': 'web',
                            'references': ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'],
                            'related_indicator_is_active': 1,
                            'related_indicator_type': 'IPv4',
                            'subscriber_count': 275,
                            'tags': ['hkcu',
                                     'class ids',
                                     'appdata',
                                     'programdata'],
                            'targeted_countries': [],
                            'threat_hunter_has_agents': 1,
                            'threat_hunter_scannable': True,
                            'upvotes_count': 0,
                            'validator_count': 0,
                            'vote': 0,
                            'votes_count': 0},
                           {'TLP': 'green',
                            'adversary': '',
                            'attack_ids': [],
                            'author': {'avatar_url': 'https://otx.alienvault.com/assets/images/default-avatar.png',
                                       'id': '157433',
                                       'is_following': False,
                                       'is_subscribed': False,
                                       'username': 'SURMI003'},
                            'cloned_from': None,
                            'comment_count': 0,
                            'created': '2021-08-03T04:46:56.557000',
                            'description': "The world's largest body of human "
                                           'evolution has a new leader - and '
                                           "it's got a lot of work to do - to "
                                           "make sure it doesn't look like it "
                                           'is going to get any worse.',
                            'downvotes_count': 0,
                            'export_count': 0,
                            'follower_count': 0,
                            'groups': [],
                            'id': '6108ca40db1c7b5f1986e2de',
                            'in_group': False,
                            'indicator_count': 0,
                            'indicator_type_counts': {},
                            'industries': [],
                            'is_author': False,
                            'is_modified': True,
                            'is_subscribing': None,
                            'locked': False,
                            'malware_families': [],
                            'modified': '2021-09-02T00:01:54.965000',
                            'modified_text': '89 days ago ',
                            'name': 'test2',
                            'public': 1,
                            'pulse_source': 'web',
                            'references': [],
                            'related_indicator_is_active': 0,
                            'related_indicator_type': 'IPv4',
                            'subscriber_count': 11,
                            'tags': [],
                            'targeted_countries': [],
                            'threat_hunter_has_agents': 1,
                            'threat_hunter_scannable': False,
                            'upvotes_count': 0,
                            'validator_count': 0,
                            'vote': 0,
                            'votes_count': 0}],
                'references': ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'],
                'related': {'alienvault': {'adversary': ['WIRTE'],
                                           'industries': ['Energy',
                                                          'Financial',
                                                          'Technology',
                                                          'Political',
                                                          'Telecommunications',
                                                          'Government',
                                                          'Military',
                                                          'Diplomatic'],
                                           'malware_families': []},
                            'other': {'adversary': [],
                                      'industries': [],
                                      'malware_families': []}}},
 'region': '12',
 'reputation': 0,
 'sections': ['general',
              'geo',
              'reputation',
              'url_list',
              'passive_dns',
              'malware',
              'nids_list',
              'http_scans'],
 'subdivision': '12',
 'type': 'IPv4',
 'type_title': 'IPv4',
 'validation': [],
 'whois': 'http://whois.domaintools.com/91.211.89.33'}

We can do the exact same set of lookups directly from that IP address if we define it as an entity:

from msticpy.datamodel.entities import IpAddress
pivot = Pivot(namespace=globals())
destip = IpAddress(Address="91.211.89.33")
ti_results = destip.tilookup_ipv4()
TILookup.browse_results(ti_results)
This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com.
This library uses services provided by ipstack. https://ipstack.com

91.211.89.33

Type: 'ipv4', Provider: OTX, severity: high

Details

OTX
pulse_count3
names['Public report on attacks in Middle East we attribute to WIRTE APT', 'WIRTE’s campaign in the Middle East ‘living off the land’ since at least 2019', 'test2']
tags[['wirte', 'macros', 'microsoft excel', 'spear phishing'], ['hkcu', 'class ids', 'appdata', 'programdata'], []]
references[['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'], ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'], []]

Reference:

https://otx.alienvault.com/api/v1/indicators/IPv4/91.211.89.33/general

Raw Results

Raw results from provider...
{'accuracy_radius': 500,
 'area_code': 0,
 'asn': 'AS206638 PE Brezhnev Daniil',
 'base_indicator': {'access_reason': '',
                    'access_type': 'public',
                    'content': '',
                    'description': '',
                    'id': 2822432629,
                    'indicator': '91.211.89.33',
                    'title': '',
                    'type': 'IPv4'},
 'charset': 0,
 'city': 'Dnipro',
 'city_data': True,
 'continent_code': 'EU',
 'country_code': 'UA',
 'country_code2': 'UA',
 'country_code3': 'UKR',
 'country_name': 'Ukraine',
 'dma_code': 0,
 'false_positive': [],
 'flag_title': 'Ukraine',
 'flag_url': '/assets/images/flags/ua.png',
 'indicator': '91.211.89.33',
 'latitude': 48.4735,
 'longitude': 35.046,
 'postal_code': '49028',
 'pulse_info': {'count': 3,
                'pulses': [{'TLP': 'white',
                            'adversary': 'WIRTE',
                            'attack_ids': [{'display_name': 'T1059 - Command '
                                                            'and Scripting '
                                                            'Interpreter',
                                            'id': 'T1059',
                                            'name': 'Command and Scripting '
                                                    'Interpreter'},
                                           {'display_name': 'T1218 - Signed '
                                                            'Binary Proxy '
                                                            'Execution',
                                            'id': 'T1218',
                                            'name': 'Signed Binary Proxy '
                                                    'Execution'},
                                           {'display_name': 'T1113 - Screen '
                                                            'Capture',
                                            'id': 'T1113',
                                            'name': 'Screen Capture'},
                                           {'display_name': 'T1546 - Event '
                                                            'Triggered '
                                                            'Execution',
                                            'id': 'T1546',
                                            'name': 'Event Triggered '
                                                    'Execution'},
                                           {'display_name': 'T1562 - Impair '
                                                            'Defenses',
                                            'id': 'T1562',
                                            'name': 'Impair Defenses'},
                                           {'display_name': 'T1036 - '
                                                            'Masquerading',
                                            'id': 'T1036',
                                            'name': 'Masquerading'},
                                           {'display_name': 'T1566 - Phishing',
                                            'id': 'T1566',
                                            'name': 'Phishing'},
                                           {'display_name': 'T1137.001 - '
                                                            'Office Template '
                                                            'Macros',
                                            'id': 'T1137.001',
                                            'name': 'Office Template Macros'},
                                           {'display_name': 'T1059.001 - '
                                                            'PowerShell',
                                            'id': 'T1059.001',
                                            'name': 'PowerShell'},
                                           {'display_name': 'T1193 - '
                                                            'Spearphishing '
                                                            'Attachment',
                                            'id': 'T1193',
                                            'name': 'Spearphishing '
                                                    'Attachment'}],
                            'author': {'avatar_url': '/otxapi/users/avatar_image/media/avatars/user_2/resized/80/avatar_dacfad0ca8.png',
                                       'id': '2',
                                       'is_following': False,
                                       'is_subscribed': True,
                                       'username': 'AlienVault'},
                            'cloned_from': None,
                            'comment_count': 0,
                            'created': '2021-11-29T16:10:35.042000',
                            'description': 'This February researchers came '
                                           'across MS Excel droppers that use '
                                           'hidden spreadsheets and VBA macros '
                                           'to drop their first stage implant. '
                                           'The implant itself is a VBS script '
                                           'with functionality to collect '
                                           'system information and execute '
                                           'arbitrary code sent by the '
                                           'attackers on the infected machine. '
                                           'Although these intrusion sets may '
                                           'appear similar to the new '
                                           'MuddyWater first stage VBS implant '
                                           'used for reconnaissance and '
                                           'profiling activities, they have '
                                           'slightly different TTPs and wider '
                                           'targeting. To date, most of the '
                                           'known victims are located in the '
                                           'Middle East, but there are also '
                                           'targets in other regions',
                            'downvotes_count': 0,
                            'export_count': 27,
                            'follower_count': 0,
                            'groups': [],
                            'id': '61a4fb7c9b88f16b103c151d',
                            'in_group': False,
                            'indicator_count': 26,
                            'indicator_type_counts': {'FileHash-MD5': 6,
                                                      'FileHash-SHA1': 2,
                                                      'FileHash-SHA256': 2,
                                                      'IPv4': 4,
                                                      'domain': 12},
                            'industries': ['Telecommunications',
                                           'Energy',
                                           'Political',
                                           'Technology',
                                           'Military',
                                           'Financial',
                                           'Diplomatic',
                                           'Government'],
                            'is_author': False,
                            'is_modified': False,
                            'is_subscribing': None,
                            'locked': False,
                            'malware_families': [],
                            'modified': '2021-11-29T16:10:35.042000',
                            'modified_text': '10 hours ago ',
                            'name': 'Public report on attacks in Middle East '
                                    'we attribute to WIRTE APT',
                            'public': 1,
                            'pulse_source': 'web',
                            'references': ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'],
                            'related_indicator_is_active': 1,
                            'related_indicator_type': 'IPv4',
                            'subscriber_count': 166586,
                            'tags': ['wirte',
                                     'macros',
                                     'microsoft excel',
                                     'spear phishing'],
                            'targeted_countries': [],
                            'threat_hunter_has_agents': 1,
                            'threat_hunter_scannable': True,
                            'upvotes_count': 0,
                            'validator_count': 0,
                            'vote': 0,
                            'votes_count': 0},
                           {'TLP': 'white',
                            'adversary': '',
                            'attack_ids': [],
                            'author': {'avatar_url': '/otxapi/users/avatar_image/media/avatars/user_94093/resized/80/avatar_281f69b768.png',
                                       'id': '94093',
                                       'is_following': False,
                                       'is_subscribed': False,
                                       'username': 'Sand-Storm'},
                            'cloned_from': None,
                            'comment_count': 0,
                            'created': '2021-11-29T17:27:20.561000',
                            'description': 'Malicious documents and droppers, '
                                           'as reported by the BBC, have been '
                                           'uncovered by researchers at the '
                                           'University of Kansas in the United '
                                           'States and are subject to an '
                                           'investigation by security firm '
                                           'Kaspersky.',
                            'downvotes_count': 0,
                            'export_count': 2,
                            'follower_count': 0,
                            'groups': [],
                            'id': '61a50d7881106990abd40bd1',
                            'in_group': False,
                            'indicator_count': 23,
                            'indicator_type_counts': {'FileHash-MD5': 6,
                                                      'FileHash-SHA1': 2,
                                                      'FileHash-SHA256': 2,
                                                      'IPv4': 4,
                                                      'domain': 9},
                            'industries': [],
                            'is_author': False,
                            'is_modified': False,
                            'is_subscribing': None,
                            'locked': False,
                            'malware_families': [],
                            'modified': '2021-11-29T17:27:20.561000',
                            'modified_text': '9 hours ago ',
                            'name': 'WIRTE’s campaign in the Middle East '
                                    '‘living off the land’ since at least 2019',
                            'public': 1,
                            'pulse_source': 'web',
                            'references': ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'],
                            'related_indicator_is_active': 1,
                            'related_indicator_type': 'IPv4',
                            'subscriber_count': 275,
                            'tags': ['hkcu',
                                     'class ids',
                                     'appdata',
                                     'programdata'],
                            'targeted_countries': [],
                            'threat_hunter_has_agents': 1,
                            'threat_hunter_scannable': True,
                            'upvotes_count': 0,
                            'validator_count': 0,
                            'vote': 0,
                            'votes_count': 0},
                           {'TLP': 'green',
                            'adversary': '',
                            'attack_ids': [],
                            'author': {'avatar_url': 'https://otx.alienvault.com/assets/images/default-avatar.png',
                                       'id': '157433',
                                       'is_following': False,
                                       'is_subscribed': False,
                                       'username': 'SURMI003'},
                            'cloned_from': None,
                            'comment_count': 0,
                            'created': '2021-08-03T04:46:56.557000',
                            'description': "The world's largest body of human "
                                           'evolution has a new leader - and '
                                           "it's got a lot of work to do - to "
                                           "make sure it doesn't look like it "
                                           'is going to get any worse.',
                            'downvotes_count': 0,
                            'export_count': 0,
                            'follower_count': 0,
                            'groups': [],
                            'id': '6108ca40db1c7b5f1986e2de',
                            'in_group': False,
                            'indicator_count': 0,
                            'indicator_type_counts': {},
                            'industries': [],
                            'is_author': False,
                            'is_modified': True,
                            'is_subscribing': None,
                            'locked': False,
                            'malware_families': [],
                            'modified': '2021-09-02T00:01:54.965000',
                            'modified_text': '89 days ago ',
                            'name': 'test2',
                            'public': 1,
                            'pulse_source': 'web',
                            'references': [],
                            'related_indicator_is_active': 0,
                            'related_indicator_type': 'IPv4',
                            'subscriber_count': 11,
                            'tags': [],
                            'targeted_countries': [],
                            'threat_hunter_has_agents': 1,
                            'threat_hunter_scannable': False,
                            'upvotes_count': 0,
                            'validator_count': 0,
                            'vote': 0,
                            'votes_count': 0}],
                'references': ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'],
                'related': {'alienvault': {'adversary': ['WIRTE'],
                                           'industries': ['Energy',
                                                          'Financial',
                                                          'Technology',
                                                          'Political',
                                                          'Telecommunications',
                                                          'Government',
                                                          'Military',
                                                          'Diplomatic'],
                                           'malware_families': []},
                            'other': {'adversary': [],
                                      'industries': [],
                                      'malware_families': []}}},
 'region': '12',
 'reputation': 0,
 'sections': ['general',
              'geo',
              'reputation',
              'url_list',
              'passive_dns',
              'malware',
              'nids_list',
              'http_scans'],
 'subdivision': '12',
 'type': 'IPv4',
 'type_title': 'IPv4',
 'validation': [],
 'whois': 'http://whois.domaintools.com/91.211.89.33'}

Its also possible to look up multiple IoCs at once from a DataFrame:

import pandas as pd
host_logons = pd.read_pickle("../data/host_logons.pkl")
ti_df = ti_lookup.lookup_iocs(host_logons, obs_col = "IpAddress")
ti_df.head()
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_52424/1394424922.py in <module>
      1 import pandas as pd
----> 2 host_logons = pd.read_pickle("data/host_logons.pkl")
      3 ti_df = ti_lookup.lookup_iocs(host_logons, obs_col = "IpAddress")
      4 ti_df.head()

~\AppData\Roaming\Python\Python37\site-packages\pandas\io\pickle.py in read_pickle(filepath_or_buffer, compression, storage_options)
    176         compression=compression,
    177         is_text=False,
--> 178         storage_options=storage_options,
    179     ) as handles:
    180 

~\AppData\Roaming\Python\Python37\site-packages\pandas\io\common.py in get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options)
    649         else:
    650             # Binary mode
--> 651             handle = open(handle, ioargs.mode)
    652         handles.append(handle)
    653 

FileNotFoundError: [Errno 2] No such file or directory: 'data/host_logons.pkl'
IpAddress.tilookup_ipv4(host_logons, input_column="IpAddress")

Its not just IP addresses either, there is support for a wide range of entity types:

from msticpy.datamodel.entities import *
md("Domains name:", "bold")
display(Dns.tilookup_dns("energyxprt.com"))
md("Files:", "bold")
display(File.tilookup_file_hash("f11750939680d28d724a73bb0830a04fa7b926aead104ad7d0d8d76df634686f"))
md("Urls:", "bold")
display(Url.tilookup_url("http://saounps.com/netfV2/932ec2a5c5a97f160704d4c6e5d32a93"))

More details on Threat Intelligence lookups in MSTICPy can be found in the documenation: https://msticpy.readthedocs.io/en/latest/data_acquisition/TIProviders.html


Enrichment - Other enrichment [Pete]

Another key feature of MSTICPy is the ability to enrich your core security log data with additional data sources that help provide additional information and context to a security analyst.
There are a number of data enrichments avaliable including:

  • GeoIP data to locate an IP Address

  • WhoIs data to provide information on a domain owner

  • Azure API data to provide additional data on Azure resources.

IP Tools

MSTICPy contains a number of IP related enrichments the are grouped under the IPAddress entity type. GeoIP is a useful feature to enrich your data with information about the location of an IP address and provide context about whether the IP address shoudl be considered suspicious or not.
MSTICPy supports getting GeoIP data from tow key sources, MaxMind’s GeoIPLite service, and the IPStack Geo service.

Note: Both of these GeoIP services require an API key - more details can be found in the MSTICPy documentation

from msticpy.datamodel.pivot import Pivot
Pivot(namespace=globals())

from msticpy.datamodel.entities import IpAddress
IpAddress.util.geoloc(value="103.125.190.248")

MSTICPy also has IP tools to get WhoIs information on an IP address:

IpAddress.util.whois(value="103.125.190.248")

And to do a resverse DNS lookup:

IpAddress.util.ip_rev_resolve(value="103.125.190.248")

Domain Tools

Similar enrichments exist for other common entity types such as domains (under the Dns entity type):

from msticpy.datamodel.entities import Dns
Dns.util.dns_resolve("www.contoso.com")
Dns.util.dns_in_abuse_list("www.contoso.com")

We can also fetch a screenshot of a target URL in order to give the analyst a visual representation of the site being investigated.

Note: Screenshots are enabled by the Browshot service

from msticpy.sectools.domain_utils import screenshot
from IPython.display import display, Image

sshot = screenshot("www.contoso.com")

with open('screenshot.png', 'wb') as f:
    f.write(sshot.content)

display(Image(filename='screenshot.png'))
from msticpy.sectools.geoip import GeoLiteLookup

iplocation = GeoLiteLookup()
loc_result, ip_entity = iplocation.lookup_ip(ip_address='90.156.201.97')
print('Raw result')
display(loc_result)

print('IP Address Entity')
display(ip_entity[0])

We can also lookup multiple IP addresses at once by passing in a list of IP addresses.

ips = ["103.125.190.248", "173.232.207.214", "52.200.40.111"]

_, ip_entities = iplocation.lookup_ip(ip_addr_list=ips)
ents = [ip_ent.properties for ip_ent in ip_entities]
pd.DataFrame(ents)

MSTICPy also has tools to support getting information such as the WhoIs record for an IP address:

from msticpy.sectools.ip_utils import get_whois_info
get_whois_info("103.125.190.248")

Azure Data Enrichment

MSTICPy also includes a number of Azure API integrations that can be used to enrich your data with additional data about Azure Resources. These are avaliable in two fromats, via the AzureData feature of MSTICPy and also via the new Azure Resource Graph data connector.

from msticpy.data.azure_data import AzureData
# Create our Azure Data instance and connect
az_data = AzureData()
az_data.connect()
# List subscriptions our account has access to 
subs = az_data.get_subscriptions()
subs.head()
Attempting to sign-in with environment variable credentials...
Attempting to sign-in with environment variable credentials...
Subscription ID Display Name State
0 8c4b5b03-3b24-4ed0-91f5-a703cd91b412 Cosmos_C&E_Azure_AzureEngineeringSystems_100200 Enabled
1 7a7b5559-58af-401a-b543-61b7321a97ea Epic-Edge-ES-GitCache-Prod Enabled
2 bac420ed-c6fc-4a05-8ac1-8c0c52da1d6e IDEAs MS Reporting Enabled
3 7fd08dcc-a653-4b0f-8f8c-4dac889fdda4 Code generate Test and Infra Enabled
4 54b875cc-a81a-4914-8bfd-1a36bc7ddf4d MSFT-WindowsVirtualDesktop-01 Enabled
subscription = subs[subs['Display Name'].str.contains("ASI Hunting Demo")].iloc[0]
subscription
Subscription ID    40dcc8bf-0478-4f3b-b275-ed0a94f2c013
Display Name               ASI Hunting Demo Environment
State                                           Enabled
Name: 84, dtype: object
# List resources in our subscription
sub_info = az_data.get_subscription_info(sub_id=subscription['Subscription ID'])
display(sub_info)
{'Subscription ID': '40dcc8bf-0478-4f3b-b275-ed0a94f2c013',
 'Display Name': 'ASI Hunting Demo Environment',
 'State': 'Enabled',
 'Subscription Location': 'Internal_2014-09-01',
 'Subscription Quota': 'Internal_2014-09-01',
 'Spending Limit': 'Off'}

We can also use the AzureSentinel feature to get details about specific Microsoft Sentitnel elements:

from msticpy.data.azure_sentinel import AzureSentinel
azure_sent = AzureSentinel()
azure_sent.connect()
Attempting to sign-in with environment variable credentials...
Attempting to sign-in with environment variable credentials...
# List our workspaces
azure_sent.get_sentinel_workspaces(sub_id=subscription['Subscription ID'])
Finding Azure Sentinel Workspaces...
Attempting to sign-in with environment variable credentials...
{'TempLAWorkspace': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/test/providers/Microsoft.OperationalInsights/workspaces/TempLAWorkspace',
 'koreasentinelworkspace': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/korearg/providers/Microsoft.OperationalInsights/workspaces/koreasentinelworkspace',
 'franceworkspace': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/france-rg/providers/Microsoft.OperationalInsights/workspaces/franceworkspace',
 'ASIHuntOMSWorkspaceV4': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/ASIHuntOMSWorkspaceRG/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4',
 'westeuroworkspace': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/westeuro-rg/providers/Microsoft.OperationalInsights/workspaces/westeuroworkspace',
 'austeastwkspc': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/ASIWorkspaceRG/providers/Microsoft.OperationalInsights/workspaces/austeastwkspc',
 'sentinellayounes': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/splunkrg/providers/Microsoft.OperationalInsights/workspaces/sentinellayounes',
 'weusworkspace': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/westusrg/providers/Microsoft.OperationalInsights/workspaces/weusworkspace',
 'weusworkspace2': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/westusrg/providers/Microsoft.OperationalInsights/workspaces/weusworkspace2'}
# Get incidents from the workspace
incidents = azure_sent.get_incidents(res_id = '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/ASIHuntOMSWorkspaceRG/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4')
incidents.head()
id name etag type properties.title properties.description properties.severity properties.status properties.owner.objectId properties.owner.email properties.owner.assignedTo properties.owner.userPrincipalName properties.labels properties.firstActivityTimeUtc properties.lastActivityTimeUtc properties.lastModifiedTimeUtc properties.createdTimeUtc properties.incidentNumber properties.additionalData.alertsCount properties.additionalData.bookmarksCount properties.additionalData.commentsCount properties.additionalData.alertProductNames properties.additionalData.tactics properties.relatedAnalyticRuleIds properties.incidentUrl
0 /subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provide... 1a070aec-12a2-49f0-ab9d-cb712b49d49a "02003475-0000-0a00-0000-618b30f10000" Microsoft.SecurityInsights/Incidents Unauthenticated access to a storage blob container Unusual unauthenticated access to your storage account 'asicsvstorage' was detected. Access to t... Medium New None None None None [] 2021-11-10T01:43:11.41Z 2021-11-10T01:43:11.41Z 2021-11-10T02:39:45.3489835Z 2021-11-10T02:39:45.3489835Z 46150 1 0 0 [Azure Security Center] [InitialAccess] [/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provid... https://portal.azure.com/#asset/Microsoft_Azure_Security_Insights/Incident/subscriptions/40dcc8b...
1 /subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provide... f4f87d6f-1755-4bf2-9e82-e929d6446bf6 "b2014a58-0000-0a00-0000-615f79010000" Microsoft.SecurityInsights/Incidents Unauthenticated access to a storage blob container Unusual unauthenticated access to your storage account 'asicsvstorage' was detected. Access to t... Medium New None None None None [] 2021-10-07T21:00:59.79Z 2021-10-07T21:00:59.79Z 2021-10-07T22:47:29.2329894Z 2021-10-07T22:47:29.2329894Z 46149 1 0 0 [Azure Security Center] [InitialAccess] [/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provid... https://portal.azure.com/#asset/Microsoft_Azure_Security_Insights/Incident/subscriptions/40dcc8b...
2 /subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provide... 24c8e377-c76a-4ee0-bb17-720768742fc2 "a201a7a1-0000-0a00-0000-615e17430000" Microsoft.SecurityInsights/Incidents PREVIEW - Suspicious management session using Azure portal detected Analysis of your subscription activity logs has detected a suspicious behavior.\nA principal tha... Medium New None None None None [] 2021-10-06T20:14:31.0557037Z 2021-10-06T20:40:13.8362148Z 2021-10-06T21:38:11.2303988Z 2021-10-06T21:38:11.2303988Z 46148 1 0 0 [Azure Security Center] [Persistence] [/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provid... https://portal.azure.com/#asset/Microsoft_Azure_Security_Insights/Incident/subscriptions/40dcc8b...
3 /subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provide... f7b36bc7-aca6-4327-8e6a-a45671c0d316 "d800ed95-0000-0a00-0000-612554db0000" Microsoft.SecurityInsights/Incidents Unusual application accessed a storage file share Someone has accessed your Azure storage account 'ianhellepub2020382608650' using an unexpected a... Medium New None None None None [] 2021-08-24T18:46:03Z 2021-08-24T18:46:03Z 2021-08-24T20:21:47.1463327Z 2021-08-24T20:21:47.1463327Z 46147 1 0 0 [Azure Security Center] [InitialAccess] [/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provid... https://portal.azure.com/#asset/Microsoft_Azure_Security_Insights/Incident/subscriptions/40dcc8b...
4 /subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provide... eed27e1a-7794-4535-b088-7463793b1748 "40004b48-0000-0a00-0000-60f601f30000" Microsoft.SecurityInsights/Incidents Access from an unusual location to a storage blob container Someone has accessed your Azure Storage account 'ngchitempaml204478910230' from an unusual locat... Low New None None None None [] 2021-07-19T21:17:57.688Z 2021-07-19T21:17:57.688Z 2021-07-19T22:51:31.3884926Z 2021-07-19T22:51:31.3884926Z 46146 1 0 0 [Azure Security Center] [InitialAccess] [/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provid... https://portal.azure.com/#asset/Microsoft_Azure_Security_Insights/Incident/subscriptions/40dcc8b...

We can also get similar details from the Azure Resource Graph. This is accessed in the same way as our other Query Providers:

res_graph_qry_prov = QueryProvider("ResourceGraph")
res_graph_qry_prov.connect()
vm_df = res_graph_qry_prov.ResourceGraph.list_virtual_machines()
vm_df.head()
Attempting to sign-in with environment variable credentials...
Attempting to sign-in with environment variable credentials...
Connected
Attempting to sign-in with environment variable credentials...
id name type tenantId kind location resourceGroup subscriptionId managedBy sku plan tags zones extendedLocation properties.provisioningState properties.storageProfile.imageReference.publisher properties.storageProfile.imageReference.exactVersion properties.storageProfile.imageReference.version properties.storageProfile.imageReference.sku properties.storageProfile.imageReference.offer properties.storageProfile.dataDisks properties.storageProfile.osDisk.name properties.storageProfile.osDisk.createOption properties.storageProfile.osDisk.diskSizeGB properties.storageProfile.osDisk.managedDisk.id ... identity.userAssignedIdentities./subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/AzSecPackAutoConfigRG/providers/Microsoft.ManagedIdentity/userAssignedIdentities/AzSecPackAutoConfigUA-eastus.clientId identity.userAssignedIdentities./subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/AzSecPackAutoConfigRG/providers/Microsoft.ManagedIdentity/userAssignedIdentities/AzSecPackAutoConfigUA-westus2.principalId identity.userAssignedIdentities./subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/AzSecPackAutoConfigRG/providers/Microsoft.ManagedIdentity/userAssignedIdentities/AzSecPackAutoConfigUA-westus2.clientId tags.testtag tags.Demo tags.Track tags.CloudPlanDate tags.Responsible tags.Deployment_Date tags.function tags.BackupGroup tags.Deployment_ID tags.department tags.BizEnv tags.type tags.Importance tags.Business_Service tags.datadog_monitored tags.SAP product tags.SAP SID properties.proximityPlacementGroup.id tags.Case tags.ISO tags.tst_vm tags.application
0 /subscriptions/3c1bb38c-82e3-4f8d-a115-a7110ba70d05/resourceGroups/CONTOSO77/providers/Microsoft... zscaler-miror-cef microsoft.compute/virtualmachines 72f988bf-86f1-41af-91ab-2d7cd011db47 westus contoso77 3c1bb38c-82e3-4f8d-a115-a7110ba70d05 None NaN NaN None None Succeeded Canonical 18.04.201907221 latest 18.04-LTS UbuntuServer [] zscaler-miror-cef_disk1_d8f5a258115c4db1bdc8483fc71e5d4f FromImage 30.0 /subscriptions/3c1bb38c-82e3-4f8d-a115-a7110ba70d05/resourceGroups/Contoso77/providers/Microsoft... ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1 /subscriptions/0c8d0493-55c3-4b3f-a0b0-c8d4d2ce0343/resourceGroups/ARC-E2E-Test-Dev-EUS2EUAP/pro... zahi-d-stage-eus2euap microsoft.compute/virtualmachines 72f988bf-86f1-41af-91ab-2d7cd011db47 eastus2euap arc-e2e-test-dev-eus2euap 0c8d0493-55c3-4b3f-a0b0-c8d4d2ce0343 None NaN NaN None None Succeeded Canonical 18.04.202111080 latest 18.04-LTS UbuntuServer [] zahi-d-stage-eus2euap_OsDisk_1_b1accaf8d5a148f8bdd69fc51400773c FromImage 30.0 /subscriptions/0c8d0493-55c3-4b3f-a0b0-c8d4d2ce0343/resourceGroups/ARC-E2E-Test-Dev-EUS2EUAP/pro... ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2 /subscriptions/3c1bb38c-82e3-4f8d-a115-a7110ba70d05/resourceGroups/contoso77/providers/Microsoft... yuval-sysmon-flow microsoft.compute/virtualmachines 72f988bf-86f1-41af-91ab-2d7cd011db47 eastus2 contoso77 3c1bb38c-82e3-4f8d-a115-a7110ba70d05 None NaN NaN [1] None Succeeded MicrosoftWindowsServer 20348.169.2108120020 latest 2022-datacenter WindowsServer [] yuval-sysmon-flow_OsDisk_1_0152ae17069a4320aa594f439a2e1668 FromImage NaN /subscriptions/3c1bb38c-82e3-4f8d-a115-a7110ba70d05/resourceGroups/Contoso77/providers/Microsoft... ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
3 /subscriptions/de5fb112-5d5d-42d4-a9ea-5f3b1359c6a6/resourceGroups/YUVALNAOR-RG/providers/Micros... yuval-sysmon-flow microsoft.compute/virtualmachines 72f988bf-86f1-41af-91ab-2d7cd011db47 eastus2 yuvalnaor-rg de5fb112-5d5d-42d4-a9ea-5f3b1359c6a6 None NaN NaN [1] None Succeeded MicrosoftWindowsServer 17763.2114.2108051826 latest 2019-Datacenter WindowsServer [] yuval-sysmon-flow_disk1_ae66aaeb62954dfbb69a09e21c81846a FromImage NaN /subscriptions/de5fb112-5d5d-42d4-a9ea-5f3b1359c6a6/resourceGroups/yuvalnaor-rg/providers/Micros... ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
4 /subscriptions/de5fb112-5d5d-42d4-a9ea-5f3b1359c6a6/resourceGroups/YEHUDABOLYPALOALTOAPI/provide... yehudabolyfirewallv9api microsoft.compute/virtualmachines 72f988bf-86f1-41af-91ab-2d7cd011db47 eastus2 yehudabolypaloaltoapi de5fb112-5d5d-42d4-a9ea-5f3b1359c6a6 None NaN NaN None None Succeeded paloaltonetworks 9.1.0 latest byol vmseries1 [] yehudabolyfirewallv9api_OsDisk_1_943495312af044f289ad3383e72b56ed FromImage 60.0 /subscriptions/de5fb112-5d5d-42d4-a9ea-5f3b1359c6a6/resourceGroups/YEHUDABOLYPALOALTOAPI/provide... ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN

5 rows × 230 columns

As with other Query Providers you can run custom queries:

query = """Resources
| where type == 'microsoft.compute/virtualmachines'
| summarize count() by PowerState = tostring(properties.extended.instanceView.powerState.code)"""
res_graph_qry_prov.exec_query(query)
PowerState count_
0 PowerState/running 1995
1 PowerState/deallocated 366
2 PowerState/stopped 25

Pivot functions [Ian]

Pivot functions are methods of entities that provide:

  • data queries related to an entity

  • enrichment functions relevant to that entity

Pivot functions are dynamically attached to entities. We created this framework to make it easier to find which functions you can use for which entity type.

Motivation

  • We had built a lot of functionality in MSTICPy for querying and enrichment

  • A lot of the functions had inconsistent type/parameter signatures

  • There was no easy discovery mechanism for these functions - you had to know

  • Using entities as pivot points is a “natural” investigation pattern

Access functionality from entities

pivot = Pivot(namespace=globals())
from msticpy.datamodel import entities

display(entities.IpAddress.whois("38.75.137.9"))
display(entities.IpAddress.geoloc("38.75.137.9"))
asn asn_cidr asn_country_code asn_date asn_description asn_registry nets nir query raw raw_referral referral
0 63023 38.75.136.0/23 US 1991-04-16 AS-GLOBALTELEHOST, US arin [{'cidr': '38.0.0.0/8', 'name': 'COGENT-A', 'handle': 'NET-38-0-0-0-1', 'range': '38.0.0.0 - 38.... None 38.75.137.9 None None None
CountryCode CountryName State City Longitude Latitude Asn TimeGenerated Type AdditionalData IpAddress
0 US United States California Los Angeles -118.2441 34.0544 None None geolocation {} 38.75.137.9
pivot.browse()

Inputs can be single values, lists or DataFrames

%%ioc --out ip_list
	SourceIP	DestinationIP	TotalBytesSent	nir	asn_registry	asn	asn_cidr	asn_country_code	asn_date	asn_description	query	nets	raw	referral	raw_referral
0	10.0.3.5	40.124.45.19	621	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1	10.16.12.1	40.124.45.19	1004	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2	10.4.5.12	13.71.172.130	247	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
3	10.4.5.12	40.77.232.95	189	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
4	10.4.5.16	13.71.172.130	46	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
5	10.4.5.16	65.55.44.109	120	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
6	10.90.78.142	104.43.212.12	12	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
7	10.90.78.71	104.43.212.12	4	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
8	20.185.182.48	38.75.137.9	8328	NaN	arin	8075
[('ipv4',
  ['10.4.5.12',
   '40.77.232.95',
   '10.90.78.71',
   '10.90.78.142',
   '65.55.44.109',
   '10.0.3.5',
   '13.71.172.130',
   '38.75.137.9',
   '40.124.45.19',
   '10.4.5.16',
   '20.185.182.48',
   '104.43.212.12',
   '10.16.12.1'])]
entities.IpAddress.whois(ip_list["ipv4"]) #, join="left")
nir asn_registry asn asn_cidr asn_country_code asn_date asn_description query nets raw referral raw_referral
0 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1 NaN arin 8075 40.76.0.0/14 US 2015-02-23 MICROSOFT-CORP-MSN-AS-BLOCK, US 40.77.232.95 [{'cidr': '40.74.0.0/15, 40.76.0.0/14, 40.125.0.0/17, 40.124.0.0/16, 40.120.0.0/14, 40.112.0.0/1... NaN NaN NaN
2 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
3 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
4 NaN arin 8075 65.52.0.0/14 US 2001-02-14 MICROSOFT-CORP-MSN-AS-BLOCK, US 65.55.44.109 [{'cidr': '65.52.0.0/14', 'name': 'MICROSOFT-1BLK', 'handle': 'NET-65-52-0-0-1', 'range': '65.52... NaN NaN NaN
5 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
6 NaN arin 8075 13.64.0.0/11 US 2015-03-26 MICROSOFT-CORP-MSN-AS-BLOCK, US 13.71.172.130 [{'cidr': '13.104.0.0/14, 13.64.0.0/11, 13.96.0.0/13', 'name': 'MSFT', 'handle': 'NET-13-64-0-0-... NaN NaN NaN
7 NaN arin 63023 38.75.136.0/23 US 1991-04-16 AS-GLOBALTELEHOST, US 38.75.137.9 [{'cidr': '38.0.0.0/8', 'name': 'COGENT-A', 'handle': 'NET-38-0-0-0-1', 'range': '38.0.0.0 - 38.... NaN NaN NaN
8 NaN arin 8075 40.124.0.0/16 US 2015-02-23 MICROSOFT-CORP-MSN-AS-BLOCK, US 40.124.45.19 [{'cidr': '40.125.0.0/17, 40.96.0.0/12, 40.112.0.0/13, 40.120.0.0/14, 40.74.0.0/15, 40.76.0.0/14... NaN NaN NaN
9 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
10 NaN arin 8075 20.184.0.0/13 US 2017-02-22 MICROSOFT-CORP-MSN-AS-BLOCK, US 20.185.182.48 [{'cidr': '20.184.0.0/13, 20.180.0.0/14', 'name': 'MSFT', 'handle': 'NET-20-180-0-0-1', 'range':... NaN NaN NaN
11 NaN arin 8075 104.40.0.0/13 US 2014-05-07 MICROSOFT-CORP-MSN-AS-BLOCK, US 104.43.212.12 [{'cidr': '104.40.0.0/13', 'name': 'MSFT', 'handle': 'NET-104-40-0-0-1', 'range': '104.40.0.0 - ... NaN NaN NaN
12 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN

Creating Pivot pipelines

list(ip_list["ipv4"])[:4]
['10.4.5.12', '40.77.232.95', '10.90.78.71', '10.90.78.142']
(
    entities.IpAddress.whois(list(ip_list["ipv4"])[:4], join="left")
    .mp_pivot.run(entities.IpAddress.geoloc, input_col="ip_column", join="left")
    .mp_pivot.run(entities.IpAddress.tilookup_ipv4, input_col="ip_column", join="left")
)

Notebooklets - “Macros” for Notebooks [Ian]

We built notebooklets because life is too short keep writing (copy/pasting) the same code over and over again.

The Notebooklets (MSTICNB) package multiple notebook cells for common investigation routines into simple functions

Repo: https://github.com/microsoft/msticnb
Docs: https://msticnb.readthedocs.io/

$ pip install msticnb

# Import and initialize MSTIC Notebooklets - companion package
# more later
import msticnb as nb
qry_prov_az.connect(WorkspaceConfig(workspace="CyberSecuritySoc"))
nb.init(query_provider=qry_prov_az)
# qry_prov_az.connect(WorkspaceConfig(workspace="CyberSecuritySoc"))

nb.browse()
host_time = nbwidgets.QueryTime()
host_time
host_summary = nb.nblts.azsent.host.HostSummary()

host_summary_rslt = host_summary.run(value="WORKSTATION6", timespan=host_time)#, options=["-bookmarks", "-azure_api"])
host_summary_rslt.browse_alerts()
host_summary_rslt.host_entity.qry_wevt_processes(start="2021-11-17 16:00", end="2021-11-17 16:20").mp_plot.timeline(group_by="Account")
host_summary_rslt.host_entity.qry_wevt_processes(start="2021-11-17 16:09", end="2021-11-17 16:10").mp_plot.process_tree(legend_col="Account")

MSTICPy Community Sprint - Jan 2022

Hackathon Logo

MSTICPy is always open to contributions from the community, whether this be fixes to the current code base, feature additions, or just new ideas and suggestions. However, we know that contributing to an Open Source project can be a bit daunting, especially if it’s not something you have done before.

To help people with this we are running a Community Sprint during January 2022.

During this sprint we are encouraging people to engage with and contribute to MSTICPy. Contributions can take any form but in order to make this as easy as possible for people we will be offering support and guidance during the month to help people work out where and how to contribute. We will provide:

  • A set of contribution ideas tailored to differing skill levels and time commitments

  • Office Hours where you can come and ask questions and get help from the MSTICPy team

  • Additional contribution resources and guidance

  • Some awesome swag for people who contribute

Want to get involved? Keep an eye on the MSTICPy GitHub page for updates on the Community Sprint - https://github.com/microsoft/msticpy/wiki/MSTICPy-January-2022-Hackathon

Or follow us on twitter for news: @ianhellen, @MSSPete, @AshwinPatil


End of Session, End of the Workshop!!!

Lunch break: 45 Minutes

By Roberto Rodriguez @Cyb3rWard0g
© Copyright 2021.