Workshop 2.4: MSTICPy#
- Microsoft Threat Intelligence Center Jupyter & Python Security Tools
Contributors:
Ian Hellen (@ianhellen)
Pete Bryan (@MSSPete)
Ashwin Patil (@ashwinpatil)
Contents
The basics
MSTICPy Widgets
Query providers revisited
Threat intelligence
Other enrichment
Pivot functions
Notebooklets - notebook macros
MSTICPy Hack month
Notebook: https://aka.ms/Jupyterthon-ws-2-4
License: Creative Commons Attribution-ShareAlike 4.0 International
Q&A - OTR Discord #Jupyterthon #WORKSHOP DAY 2 - MSTICPy
MSTICPy is a library for InfoSec investigation and hunting in Jupyter Notebooks. It includes functionality to:
query log data from multiple sources
enrich the data with Threat Intelligence, geolocations and Azure resource data
extract Indicators of Activity (IoA) from logs and unpack encoded data
perform sophisticated analysis such as anomalous session detection and time series decomposition
visualize data using interactive timelines, process trees and multi-dimensional Morph Charts
It also includes some time-saving notebook tools such as widgets to set query time boundaries, select and display items from lists, and configure the notebook environment.
Source Code: microsoft/msticpy Python Package: https://pypi.org/project/msticpy/#:~:text=Microsoft Threat Intelligence Python Security Tools. msticpy is,functionality to%3A query log data from multiple sources Docs: https://msticpy.readthedocs.io/en/latest/
Basics - installing and configuring#
To use any library in Python you first need to install the pacakge and import it. There are several ways to do this depending on how you want to access the library, however the simplest and easiest is using pip. Pip is the pacakge installer for Python and makes finding and installing Python pacakges simple. You can use pip to install packages via the command line, or if you are using a notebook, directly in a notebook cell. Azure ML compute come with Pip installed already but if you are running your notebook elsewhere you may need to install pip first.
To do this we need to use %pip followed by install and the pacakge name. e.g.:
%pip install msticpy
MSTICPy is a library with a broad range of functionality. As such installing the whole library can be more than required for a lot of uses. As such MSTICPy has implemented a series of Extras that allow for the installation of certain part of the library. These Extras are grouped around core technologies that you might want to use with MSTICPy.
Extra |
Functionality |
|---|---|
–none– |
Most functionality (approx 75%) Kqlmagic Jupyter basic |
keyvault |
Key Vault and keyring storage of settings secrets |
azure |
Azure API data retrieval, Azure storage APIs, Sentinel APIs |
kql |
Kqlmagic Jupyter extended functionality |
azsentinel |
Combination of core install + “azure”, “keyvault”, “kql” |
ml |
Timeseries analysis, Event clustering, Outlier analysis |
splunk |
Splunk data queries |
vt3 |
VirusTotal V3 graph API |
riskiq |
RiskIQ Illuminate threat intel provider & pivot functions |
all |
Includes all of above packages |
dev |
Development tools plus “base” |
test |
“dev” plus “all” |
To install a specific Extra, use the following syntax:
%pip install msticpy[extra]
You can also install multiple extras at once:
%pip install msticpy[extra1,extra2,...]
%pip install msticpy[all]
Requirement already satisfied: msticpy[all] in e:\src\microsoft\msticpy (1.5.0)
Requirement already satisfied: attrs>=18.2.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (21.2.0)
Requirement already satisfied: azure-common>=1.1.18 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.1.27)
Requirement already satisfied: azure-core>=1.2.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.19.0)
Requirement already satisfied: azure-identity>=1.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.5.0)
Requirement already satisfied: azure-mgmt-subscription>=1.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.0.0)
Requirement already satisfied: bokeh>=1.4.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.3.2)
Requirement already satisfied: cryptography>=3.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (3.4.8)
Requirement already satisfied: deprecated>=1.2.4 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.2.10)
Requirement already satisfied: dnspython<=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.16.0)
Requirement already satisfied: folium>=0.9.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.11.0)
Requirement already satisfied: geoip2>=2.9.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (4.1.0)
Requirement already satisfied: html5lib in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.1)
Requirement already satisfied: idna<3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.10)
Requirement already satisfied: ipwhois>=1.1.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.2.0)
Requirement already satisfied: ipython>=7.1.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (7.27.0)
Requirement already satisfied: ipywidgets>=7.4.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (7.6.4)
Requirement already satisfied: KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.1.114.dev26)
Requirement already satisfied: lxml>=4.6.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (4.6.3)
Requirement already satisfied: matplotlib>=3.0.0 in c:\users\ian\appdata\roaming\python\python37\site-packages (from msticpy[all]) (3.3.3)
Requirement already satisfied: msrest>=0.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.6.21)
Requirement already satisfied: msrestazure>=0.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.6.4)
Requirement already satisfied: networkx>=2.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.6.3)
Requirement already satisfied: numpy>=1.15.4 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.21.2)
Requirement already satisfied: pandas>=1.1.5 in c:\users\ian\appdata\roaming\python\python37\site-packages (from msticpy[all]) (1.2.1)
Requirement already satisfied: python-dateutil>=2.8.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.8.1)
Requirement already satisfied: pytz>=2019.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2020.1)
Requirement already satisfied: pyyaml>=3.13 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (5.4.1)
Requirement already satisfied: requests>=2.21.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.26.0)
Requirement already satisfied: setuptools>=40.6.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (54.1.2)
Requirement already satisfied: tldextract>=2.2.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (3.0.2)
Requirement already satisfied: tqdm>=4.36.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (4.60.0)
Requirement already satisfied: urllib3>=1.23 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.25.10)
Requirement already satisfied: azure-keyvault-secrets>=4.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (4.2.0)
Requirement already satisfied: azure-mgmt-compute>=4.6.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (17.0.0)
Requirement already satisfied: azure-mgmt-core>=1.2.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.3.0)
Requirement already satisfied: azure-mgmt-keyvault>=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (9.1.0)
Requirement already satisfied: azure-mgmt-monitor>=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.0.0)
Requirement already satisfied: azure-mgmt-network>=2.7.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (16.0.0)
Requirement already satisfied: azure-mgmt-resource>=16.1.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (20.0.0)
Requirement already satisfied: azure-mgmt-resourcegraph>=8.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (8.0.0)
Requirement already satisfied: azure-storage-blob>=12.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (12.9.0)
Requirement already satisfied: keyring>=13.2.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (23.2.1)
Requirement already satisfied: moz_sql_parser<=4.11.21016,>=4.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (4.9.21002)
Requirement already satisfied: nest_asyncio>=1.4.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.5.1)
Requirement already satisfied: openpyxl>=3.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (3.0.5)
Requirement already satisfied: passivetotal>=2.5.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (2.5.3)
Requirement already satisfied: scikit-learn>=0.20.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.23.0)
Requirement already satisfied: scipy>=1.1.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.5.4)
Requirement already satisfied: splunk-sdk>=1.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.6.14)
Requirement already satisfied: statsmodels>=0.11.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.12.0)
Requirement already satisfied: sumologic-sdk>=0.1.11 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.1.11)
Requirement already satisfied: vt-graph-api>=1.0.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (1.0.1)
Requirement already satisfied: vt-py>=0.6.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msticpy[all]) (0.6.1)
Requirement already satisfied: six>=1.11.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from azure-core>=1.2.2->msticpy[all]) (1.16.0)
Requirement already satisfied: msal-extensions~=0.3.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from azure-identity>=1.5.0->msticpy[all]) (0.3.0)
Requirement already satisfied: msal<2.0.0,>=1.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from azure-identity>=1.5.0->msticpy[all]) (1.11.0)
Requirement already satisfied: packaging>=16.8 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from bokeh>=1.4.0->msticpy[all]) (21.0)
Requirement already satisfied: pillow>=7.1.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from bokeh>=1.4.0->msticpy[all]) (8.3.1)
Requirement already satisfied: tornado>=5.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from bokeh>=1.4.0->msticpy[all]) (6.1)
Requirement already satisfied: typing-extensions>=3.7.4 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from bokeh>=1.4.0->msticpy[all]) (3.10.0.2)
Requirement already satisfied: Jinja2>=2.9 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from bokeh>=1.4.0->msticpy[all]) (3.0.1)
Requirement already satisfied: cffi>=1.12 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from cryptography>=3.1->msticpy[all]) (1.14.6)
Requirement already satisfied: pycparser in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from cffi>=1.12->cryptography>=3.1->msticpy[all]) (2.20)
Requirement already satisfied: wrapt<2,>=1.10 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from deprecated>=1.2.4->msticpy[all]) (1.12.1)
Requirement already satisfied: branca>=0.3.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from folium>=0.9.0->msticpy[all]) (0.4.2)
Requirement already satisfied: maxminddb<3.0.0,>=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from geoip2>=2.9.0->msticpy[all]) (2.0.3)
Requirement already satisfied: aiohttp<4.0.0,>=3.6.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from geoip2>=2.9.0->msticpy[all]) (3.6.3)
Requirement already satisfied: async-timeout<4.0,>=3.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from aiohttp<4.0.0,>=3.6.2->geoip2>=2.9.0->msticpy[all]) (3.0.1)
Requirement already satisfied: multidict<5.0,>=4.5 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from aiohttp<4.0.0,>=3.6.2->geoip2>=2.9.0->msticpy[all]) (4.7.6)
Requirement already satisfied: yarl<1.6.0,>=1.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from aiohttp<4.0.0,>=3.6.2->geoip2>=2.9.0->msticpy[all]) (1.5.1)
Requirement already satisfied: chardet<4.0,>=2.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from aiohttp<4.0.0,>=3.6.2->geoip2>=2.9.0->msticpy[all]) (3.0.4)
Requirement already satisfied: traitlets>=4.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (5.1.0)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (3.0.20)
Requirement already satisfied: pickleshare in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (0.7.5)
Requirement already satisfied: pygments in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (2.10.0)
Requirement already satisfied: colorama in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (0.4.4)
Requirement already satisfied: matplotlib-inline in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (0.1.2)
Requirement already satisfied: jedi>=0.16 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (0.18.0)
Requirement already satisfied: decorator in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (5.1.0)
Requirement already satisfied: backcall in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipython>=7.1.1->msticpy[all]) (0.2.0)
Requirement already satisfied: jupyterlab-widgets>=1.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipywidgets>=7.4.2->msticpy[all]) (1.0.0)
Requirement already satisfied: nbformat>=4.2.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipywidgets>=7.4.2->msticpy[all]) (5.1.3)
Requirement already satisfied: ipython-genutils~=0.2.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipywidgets>=7.4.2->msticpy[all]) (0.2.0)
Requirement already satisfied: widgetsnbextension~=3.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipywidgets>=7.4.2->msticpy[all]) (3.5.1)
Requirement already satisfied: ipykernel>=4.5.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipywidgets>=7.4.2->msticpy[all]) (6.4.1)
Requirement already satisfied: argcomplete>=1.12.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (1.12.3)
Requirement already satisfied: jupyter-client<8.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (7.0.1)
Requirement already satisfied: debugpy<2.0,>=1.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (1.4.1)
Requirement already satisfied: importlib-metadata<5 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (4.8.1)
Requirement already satisfied: zipp>=0.5 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from importlib-metadata<5->ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (3.6.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from jedi>=0.16->ipython>=7.1.1->msticpy[all]) (0.8.1)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from Jinja2>=2.9->bokeh>=1.4.0->msticpy[all]) (2.0.1)
Requirement already satisfied: pyzmq>=13 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (22.2.1)
Requirement already satisfied: entrypoints in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (0.3)
Requirement already satisfied: jupyter-core>=4.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (4.8.1)
Requirement already satisfied: pywin32>=1.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from jupyter-core>=4.6.0->jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets>=7.4.2->msticpy[all]) (228)
Requirement already satisfied: pywin32-ctypes!=0.1.0,!=0.1.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from keyring>=13.2.1->msticpy[all]) (0.2.0)
Requirement already satisfied: prettytable>=0.7.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (0.7.2)
Requirement already satisfied: pyperclip>=1.7.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (1.8.0)
Requirement already satisfied: beautifulsoup4>=4.6.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (4.9.3)
Requirement already satisfied: password-strength>=0.0.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (0.0.3.post2)
Requirement already satisfied: Markdown>=3.0.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (3.3.4)
Requirement already satisfied: flask>=1.0.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (1.1.2)
Requirement already satisfied: psutil>=5.4.7 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (5.7.2)
Requirement already satisfied: isodate>=0.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (0.6.0)Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: soupsieve>1.2 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from beautifulsoup4>=4.6.3->KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (2.2.1)
Requirement already satisfied: click>=5.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from flask>=1.0.3->KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (7.1.2)
Requirement already satisfied: Werkzeug>=0.15 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from flask>=1.0.3->KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (1.0.1)
Requirement already satisfied: itsdangerous>=0.24 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from flask>=1.0.3->KqlmagicCustom[auth_code_clipboard,jupyter-basic]>=0.1.114.dev26->msticpy[all]) (1.1.0)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from matplotlib>=3.0.0->msticpy[all]) (1.3.1)
Requirement already satisfied: cycler>=0.10 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from matplotlib>=3.0.0->msticpy[all]) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from matplotlib>=3.0.0->msticpy[all]) (2.4.7)
Requirement already satisfied: mo-dots==4.2.20340 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from moz_sql_parser<=4.11.21016,>=4.5.0->msticpy[all]) (4.2.20340)
Requirement already satisfied: mo-future==3.147.20327 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from moz_sql_parser<=4.11.21016,>=4.5.0->msticpy[all]) (3.147.20327)
Requirement already satisfied: mo-logs==4.3.20340 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from moz_sql_parser<=4.11.21016,>=4.5.0->msticpy[all]) (4.3.20340)
Requirement already satisfied: mo-imports==3.149.20327 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from mo-dots==4.2.20340->moz_sql_parser<=4.11.21016,>=4.5.0->msticpy[all]) (3.149.20327)
Requirement already satisfied: mo-kwargs==3.93.20259 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from mo-logs==4.3.20340->moz_sql_parser<=4.11.21016,>=4.5.0->msticpy[all]) (3.93.20259)
Requirement already satisfied: PyJWT[crypto]<3,>=1.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msal<2.0.0,>=1.6.0->azure-identity>=1.5.0->msticpy[all]) (1.7.1)
Requirement already satisfied: portalocker~=1.6 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msal-extensions~=0.3.0->azure-identity>=1.5.0->msticpy[all]) (1.7.1)
Requirement already satisfied: requests-oauthlib>=0.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msrest>=0.6.0->msticpy[all]) (1.3.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msrest>=0.6.0->msticpy[all]) (2021.10.8)
Requirement already satisfied: adal<2.0.0,>=0.6.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from msrestazure>=0.6.0->msticpy[all]) (1.2.4)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbformat>=4.2.0->ipywidgets>=7.4.2->msticpy[all]) (3.2.0)
Requirement already satisfied: pyrsistent>=0.14.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.4.2->msticpy[all]) (0.17.3)
Requirement already satisfied: et-xmlfile in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from openpyxl>=3.0->msticpy[all]) (1.0.1)
Requirement already satisfied: jdcal in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from openpyxl>=3.0->msticpy[all]) (1.4.1)
Requirement already satisfied: future in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from passivetotal>=2.5.3->msticpy[all]) (0.18.2)
Requirement already satisfied: ez-setup in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from passivetotal>=2.5.3->msticpy[all]) (0.9)
Requirement already satisfied: wcwidth in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=7.1.1->msticpy[all]) (0.2.5)
Requirement already satisfied: charset-normalizer~=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from requests>=2.21.1->msticpy[all]) (2.0.4)
Requirement already satisfied: oauthlib>=3.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from requests-oauthlib>=0.5.0->msrest>=0.6.0->msticpy[all]) (3.0.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from scikit-learn>=0.20.2->msticpy[all]) (2.1.0)
Requirement already satisfied: joblib>=0.11 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from scikit-learn>=0.20.2->msticpy[all]) (1.0.1)
Requirement already satisfied: patsy>=0.5 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from statsmodels>=0.11.1->msticpy[all]) (0.5.1)
Requirement already satisfied: requests-file>=1.4 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from tldextract>=2.2.2->msticpy[all]) (1.5.1)
Requirement already satisfied: filelock>=3.0.8 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from tldextract>=2.2.2->msticpy[all]) (3.0.12)
Requirement already satisfied: futures in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from vt-graph-api>=1.0.1->msticpy[all]) (3.1.1)
Requirement already satisfied: notebook>=4.4.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (6.4.3)
Requirement already satisfied: Send2Trash>=1.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (1.8.0)
Requirement already satisfied: terminado>=0.8.3 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.9.4)
Requirement already satisfied: argon2-cffi in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (20.1.0)
Requirement already satisfied: prometheus-client in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.11.0)
Requirement already satisfied: nbconvert in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (6.0.7)
Requirement already satisfied: pywinpty>=0.5 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from terminado>=0.8.3->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.5.7)
Requirement already satisfied: webencodings in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from html5lib->msticpy[all]) (0.5.1)
Requirement already satisfied: defusedxml in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.7.1)
Requirement already satisfied: jupyterlab-pygments in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.1.2)
Requirement already satisfied: testpath in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.5.0)
Requirement already satisfied: bleach in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (4.0.0)
Requirement already satisfied: pandocfilters>=1.4.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (1.4.3)
Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.5.3)
Requirement already satisfied: mistune<2,>=0.8.1 in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (0.8.4)
Requirement already satisfied: async-generator in c:\users\ian\anaconda3\envs\condadev\lib\site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.4.2->msticpy[all]) (1.10)
Configuration#
Once installed MSTICPy can be imported in the same way as any other Python library, however to make things a bit easier we have created the init_notebook function that will automatically import the library and configure it for use in a Jupyter Notebook.
Note: Passing `globals()` lets the init function import stuff into the notebook top-level namespace
import msticpy
msticpy.init_notebook(globals())
Starting Notebook initialization...
msticpy version installed: 1.5.0rc3 latest published: 1.5.0
A newer version of msticpy - 1.5.0 is available.
Upgrade with pip install --upgrade msticpy
A newer version of msticpy - 1.5.0 is available.
Upgrade with pip install --upgrade msticpy
Processing imports....
Imported: pd (pandas), IPython.get_ipython, IPython.display.display, IPython.display.HTML, IPython.display.Markdown, widgets (ipywidgets), pathlib.Path, plt (matplotlib.pyplot), matplotlib.MatplotlibDeprecationWarning, sns (seaborn), np (numpy), msticpy, msticpy.data.QueryProvider, msticpy.nbtools.foliummap.FoliumMap, msticpy.common.utility.md, msticpy.common.utility.md_warn, msticpy.common.wsconfig.WorkspaceConfig, msticpy.datamodel.pivot.Pivot, msticpy.datamodel.entities, msticpy.vis.mp_pandas_plot
Checking configuration....
No valid configuration for Azure Sentinel found.
Azure CLI not detected. AzCLI single sign-on disabled ({_CLI_WIKI_MSSG_SHORT})
Setting notebook options....
Notebook setup completed with some warnings.
One or more configuration items were missing or set incorrectly.
Please run the Getting Started Guide for Azure Sentinel ML Notebooks notebook. and the msticpy configuration guide.
This notebook may still run but with reduced functionality.
Notebook initialization complete
False
MSTICPy’s config file#
MSTICPy can handle connections to a variety of data sources and services, including Azure Sentinel.
To make it easier to manage and re-use the configuration and credentials fo these things MSTICPy has its own config file that holds these items - msticpyconfig.yaml
If you didn’t have a msticpyconfig.yaml file in your workspace folder (which is likely if this is your first use of MSTICPY), the init_notebook function should have created one for you and populated it file structure you are in.
You can populate msticpyconfig manually or you can used MSTICPy’s settings editor to view and edit the settings stored there.
msticpy.MpConfigEdit()
There are different settings depending on what feature of MSTICPy you are setting configuration for. This includes config settings for connecting to Security data sources such as Microsoft Sentinel, Threat Intelligence providers such as VirusTotal, or for connecting to KeyVault if you want to use that service for securely storing your MSTICPy settings.
Once you have completed a these sections you can check that you have valid settings by:
using the Validate Settings button
using the
msticpy.settings.settingsattribute to display current settings.
Tip: If you edit the settings, call msticpy.settings.refresh_config() to reload the settings.
import yaml
print(yaml.safe_dump(msticpy.settings.settings)[:500])
DataProviders: {}
QueryDefinitions:
Default:
- queries
msticpy:
FriendlyExceptions: true
More details on populating the config file can be found in the MSTICPy Settings Editor Documentation
MSTICPy Widgets#
MSTICPy include a series of Notebook widgets to make interacting with data easier, especially for users without a programming background.
The widgets are designed to fulfil a number of common tasks that a user might need to interact with a notebook such as select items from returned data, or set a timeframe for a query.
The widgets themselves are build in ipywidgets and are available in the msticpy.nbtools.nbwidgets module.
Note: Widgets are automatically imported by init_notebook
The below code creates our Time Range widget that can be used to allow a user to set a time range. We are telling it to use days as its unit of measurements and set a max range to select from.
from msticpy.nbtools.nbwidgets import *
time_select = QueryTime(units="day", max_before=20, before=5, max_after=1)
time_select.display()
We can then call the start end end properties and get datetime objects based on the user selection.
time_select.start
datetime.datetime(2021, 11, 26, 18, 22, 2, 56675)
Other widgets allow for the selection of items from list, along with a text filter option to help users find items:
items = ["item 1", "item 2", "item 3"]
selection = SelectItem(item_list=items, description="Select item", auto_display=True)
There are also security specific widgets such as SelectAlert which allows a user to select a specific alert from a list of alerts. With this widget and others you can also specify an action, this is a follow on funciton that is executed with the value of the value selected in the widget.
In the cell below we are using the action method to display the selected alert.
import pandas as pd
from msticpy.nbtools.nbdisplay import display_alert
# As discussed earlier pands read_* functions can call remote files as well as local ones.
alerts = pd.read_pickle("https://github.com/microsoft/msticpy/raw/main/tests/testdata/localdata/alerts_list.pkl")
alert_select = SelectAlert(alerts=alerts, action=display_alert)
alert_select.display()
Selected Alert: 'SSH Anomalous Login ML'
Alert_time: 2019-02-18 01:49:02, Compr_entity: , Alert_id: f1ce87ca-8863-4a66-a0bd-a4d3776a7c64| 0 | |
|---|---|
| TenantId | 52b1ab41-869e-4138-9e40-2a4457f09bf0 |
| TimeGenerated | 2019-02-18 02:29:07 |
| AlertDisplayName | SSH Anomalous Login ML |
| AlertName | SSH Anomalous Login ML |
| Severity | Low |
| Description | Anomalous login detected for SSH account |
| ProviderName | CustomAlertRule |
| VendorName | Alert Rule |
| VendorOriginalId | b0e143b8-4fa8-47bc-8bc1-9780c8b75541 |
| SystemAlertId | f1ce87ca-8863-4a66-a0bd-a4d3776a7c64 |
| ResourceId | |
| SourceComputerId | |
| AlertType | CustomAlertRule_0a4e5f7c-9756-45f8-83c4-94c756844698 |
| ConfidenceLevel | Unknown |
| ConfidenceScore | NaN |
| IsIncident | False |
| StartTimeUtc | 2019-02-18 01:49:02 |
| EndTimeUtc | 2019-02-18 02:19:02 |
| ProcessingEndTime | 2019-02-18 02:29:07 |
| RemediationSteps | |
| ExtendedProperties | {'Alert Mode': 'Aggregated', 'Search Query': '{"detailBladeInputs":{"id":"/subscriptions/40dcc8b... |
| Entities | [{'$id': '3', 'Address': '23.97.60.214', 'Type': 'ip', 'Count': 1}, {'$id': '4', 'HostName': 'MS... |
| SourceSystem | Detection |
| WorkspaceSubscriptionId | 40dcc8bf-0478-4f3b-b275-ed0a94f2c013 |
| WorkspaceResourceGroup | asihuntomsworkspacerg |
| ExtendedLinks | |
| ProductName | |
| ProductComponentName | |
| AlertLink | |
| Type | SecurityAlert |
| CompromisedEntity |
Some widgets are registered, meaning that they can be re-used later and will rememeber previous values entered.
This can be done by simply creating a widget with the same parameters as previously.
mem_text = GetText(prompt="Enter your name")
mem_text
mem_text2 = GetText(prompt="Enter your name")
mem_text2
Other MSTICPy widgets include:
A simple datetime based lookback slider
LookbackA text box to capture user input
GetTextA widget to capture and return an Environemnt Variable
GetEnvrionmentKeyA widget to select a subset of items from a list
SelectSubsetA widget to show progress of a task
ProgressMulti option buttons with a wait function that pauses cell execution until a user selects an option
OptionButtons
More details on MSTICPy’s widgets can be found here: https://msticpy.readthedocs.io/en/latest/visualization/NotebookWidgets.html
Query providers revisited [Ian]#
Supported providers#
Microsoft Sentinel
Microsoft Defender/Defender for Endpoint
Splunk
Sumologic
Microsoft Graph
Local data
Mordor/Security Datasets
Kusto/Azure Data Explorer
Azure Resource Graph
from msticpy.data import QueryProvider
import pandas as pd
# Load query providers (typically you'll be using just one)
qry_prov_az = QueryProvider("AzureSentinel")
qry_prov_sp = QueryProvider("Splunk")
qry_prov_mde = QueryProvider("MDE")
# Special provider that uses local data files
qry_prov_loc = QueryProvider("LocalData", data_paths=["./data"], query_paths=["./data"])
Please wait. Loading Kqlmagic extension...done
Warning: Custom query definitions path data not found
list_queries and running a query#
qry_prov_az.list_queries()[:10]
['Azure.get_vmcomputer_for_host',
'Azure.get_vmcomputer_for_ip',
'Azure.list_aad_signins_for_account',
'Azure.list_aad_signins_for_ip',
'Azure.list_all_signins_geo',
'Azure.list_azure_activity_for_account',
'Azure.list_azure_activity_for_ip',
'Azure.list_azure_activity_for_resource',
'Azure.list_storage_ops_for_hash',
'Azure.list_storage_ops_for_ip']
qry_prov_az.Azure.list_aad_signins_for_account?
Signature: qry_prov_az.Azure.list_aad_signins_for_account(*args, **kwargs) -> Union[pandas.core.frame.DataFrame, Any]
Call signature: qry_prov_az.Azure.list_aad_signins_for_account(*args, **kwargs)
Type: partial
String form: functools.partial(<bound method QueryProvider._execute_query of <msticpy.data.data_providers.Quer <...> er object at 0x000001F0BCA131C8>>, query_path='Azure', query_name='list_aad_signins_for_account')
File: c:\users\ian\anaconda3\envs\condadev\lib\functools.py
Docstring:
Lists Azure AD Signins for Account
Parameters
----------
account_name: str
The account name to find
add_query_items: str (optional)
Additional query clauses
end: datetime (optional)
Query end time
start: datetime (optional)
Query start time
(default value is: -5)
table: str (optional)
Table name
(default value is: SigninLogs)
Class docstring:
partial(func, *args, **keywords) - new function with partial application
of the given arguments and keywords.
Query time ranges#
Each provider has a built-in (modifiable) time range.
This supplies time parameters:
If the query requires them
You have not overridden them
The query template does not provide its own default values
qry_prov_az.query_time
Query parameters#
qry_prov_az.browse()
Lists Azure Activity for Caller IP Address(es)
Parameters
add_query_items: str (optional)
Additional query clauses
end: datetime (optional)
Query end time
ip_address_list: list
The IP Address or list of Addresses
start: datetime (optional)
Query start time
(default value is: -5)
table: str (optional)
Table name
(default value is: AzureActivity)
Query
{table}
| where CallerIpAddress in ({ip_address_list})
| where TimeGenerated >= datetime({start})
| where TimeGenerated <= datetime({end}) {add_query_items}Example
{QueryProvider}[.QueryPath].QueryName(params...)
qry_prov.Azure.list_azure_activity_for_ip(start=start, end=end, hostname=host)
{table}
| where CallerIpAddress in ({ip_address_list})
| where TimeGenerated >= datetime({start})
| where TimeGenerated <= datetime({end})
{add_query_items}
| where CallerIpAddress in ({ip_address_list})
| where TimeGenerated >= datetime({start})
| where TimeGenerated <= datetime({end})
{add_query_items}
qry_prov_az.connect(WorkspaceConfig("CyberSecuritySOC"))
qry_prov_az.Azure.list_azure_activity_for_ip()
| OperationName | OperationNameValue | Level | ActivityStatus | ActivityStatusValue | ActivitySubstatus | ActivitySubstatusValue | ResourceGroup | SubscriptionId | CorrelationId | Caller | CallerIpAddress | Category | CategoryValue | HTTPRequest | Properties | EventSubmissionTimestamp | Authorization | ResourceId | OperationId | ResourceProvider | ResourceProviderValue | Resource | EventDataId | TenantId | TimeGenerated | SourceSystem | Authorization_d | Claims | Claims_d | Properties_d | Hierarchy | Type | _ResourceId |
|---|
qry_prov_az.Azure.list_azure_activity_for_ip(
ip_address_list="176.199.184.128", add_query_items="| take 3"
)
| OperationName | OperationNameValue | Level | ActivityStatus | ActivityStatusValue | ActivitySubstatus | ActivitySubstatusValue | ResourceGroup | SubscriptionId | CorrelationId | Caller | CallerIpAddress | Category | CategoryValue | HTTPRequest | Properties | EventSubmissionTimestamp | Authorization | ResourceId | OperationId | ResourceProvider | ResourceProviderValue | Resource | EventDataId | TenantId | TimeGenerated | SourceSystem | Authorization_d | Claims | Claims_d | Properties_d | Hierarchy | Type | _ResourceId | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Check user authorization and license | Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action | Informational | Started | Started | soc | d1d8779d-38d7-4f06-91db-9cbc8de0176f | 2aaa092a-7b3c-4a77-a056-57ae1a59f7aa | Markus.Falkowski@microsoft.com | 176.199.184.128 | Administrative | Administrative | {\r\n "clientRequestId": "8bdab8e4-50bb-4108-a814-1baa55ebe019",\r\n "clientIpAddress": "176.1... | {\r\n "eventCategory": "Administrative",\r\n "entity": "/subscriptions/d1d8779d-38d7-4f06-91db... | 2021-11-29 11:22:17.142000+00:00 | {\r\n "action": "Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action",\r\n "scop... | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourceGroups/soc/providers/Microsoft.Opera... | 2ddbaaaa-246a-4fcc-aa54-3081fee17dbc | Microsoft.SecurityInsights | Microsoft.SecurityInsights | Microsoft.SecurityInsights | c682a3e7-7162-4a65-bdbf-98b54a6374b1 | 8ecf8077-cf51-4820-aadd-14040956f35d | 2021-11-29 11:20:14.994000+00:00 | Azure | None | None | None | AzureActivity | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourcegroups/soc/providers/microsoft.opera... | ||||
| 1 | Check user authorization and license | Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action | Informational | Started | Started | soc | d1d8779d-38d7-4f06-91db-9cbc8de0176f | 2aaa092a-7b3c-4a77-a056-57ae1a59f7aa | Markus.Falkowski@microsoft.com | 176.199.184.128 | Administrative | Administrative | {\r\n "clientRequestId": "8bdab8e4-50bb-4108-a814-1baa55ebe019",\r\n "clientIpAddress": "176.1... | {\r\n "eventCategory": "Administrative",\r\n "entity": "/subscriptions/d1d8779d-38d7-4f06-91db... | 2021-11-29 11:21:33.173000+00:00 | {\r\n "action": "Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action",\r\n "scop... | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourceGroups/soc/providers/Microsoft.Opera... | 945c61d7-e243-4ced-b5b8-66e1e93f7369 | Microsoft.SecurityInsights | Microsoft.SecurityInsights | Microsoft.SecurityInsights | 529832af-de9a-40d9-b4d1-a82a5da34302 | 8ecf8077-cf51-4820-aadd-14040956f35d | 2021-11-29 11:20:14.995000+00:00 | Azure | None | None | None | AzureActivity | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourcegroups/soc/providers/microsoft.opera... | ||||
| 2 | Check user authorization and license | Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action | Informational | Started | Started | soc | d1d8779d-38d7-4f06-91db-9cbc8de0176f | 2aaa092a-7b3c-4a77-a056-57ae1a59f7aa | Markus.Falkowski@microsoft.com | 176.199.184.128 | Administrative | Administrative | {\r\n "clientRequestId": "8bdab8e4-50bb-4108-a814-1baa55ebe019",\r\n "clientIpAddress": "176.1... | {\r\n "eventCategory": "Administrative",\r\n "entity": "/subscriptions/d1d8779d-38d7-4f06-91db... | 2021-11-29 11:21:39.153000+00:00 | {\r\n "action": "Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action",\r\n "scop... | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourceGroups/soc/providers/Microsoft.Opera... | 9cb7dbe5-e3f7-4dad-95e8-b46da3d4ac72 | Microsoft.SecurityInsights | Microsoft.SecurityInsights | Microsoft.SecurityInsights | fcbe3a89-a8d2-4796-aa38-70dfa32461f8 | 8ecf8077-cf51-4820-aadd-14040956f35d | 2021-11-29 11:20:15.001000+00:00 | Azure | None | None | None | AzureActivity | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourcegroups/soc/providers/microsoft.opera... |
DateTime parameters are special#
Can supply:
Python datetimes
Date string
integers
0 = now
negative numbers == # of days (or partial days) before now
positive numbers == # of days (or partial days) after now
from datetime import datetime
str(datetime.utcnow())
'2021-11-30 03:15:57.033469'
from datetime import datetime
qry_prov_az.Azure.list_azure_activity_for_ip(
ip_address_list="176.199.184.128",
start=-1.5,
end=str(datetime.utcnow()),
add_query_items="| take 3"
)
| OperationName | OperationNameValue | Level | ActivityStatus | ActivityStatusValue | ActivitySubstatus | ActivitySubstatusValue | ResourceGroup | SubscriptionId | CorrelationId | Caller | CallerIpAddress | Category | CategoryValue | HTTPRequest | Properties | EventSubmissionTimestamp | Authorization | ResourceId | OperationId | ResourceProvider | ResourceProviderValue | Resource | EventDataId | TenantId | TimeGenerated | SourceSystem | Authorization_d | Claims | Claims_d | Properties_d | Hierarchy | Type | _ResourceId | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Check user authorization and license | Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action | Informational | Started | Started | soc | d1d8779d-38d7-4f06-91db-9cbc8de0176f | 2aaa092a-7b3c-4a77-a056-57ae1a59f7aa | Markus.Falkowski@microsoft.com | 176.199.184.128 | Administrative | Administrative | {\r\n "clientRequestId": "8bdab8e4-50bb-4108-a814-1baa55ebe019",\r\n "clientIpAddress": "176.1... | {\r\n "eventCategory": "Administrative",\r\n "entity": "/subscriptions/d1d8779d-38d7-4f06-91db... | 2021-11-29 11:22:17.142000+00:00 | {\r\n "action": "Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action",\r\n "scop... | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourceGroups/soc/providers/Microsoft.Opera... | 2ddbaaaa-246a-4fcc-aa54-3081fee17dbc | Microsoft.SecurityInsights | Microsoft.SecurityInsights | Microsoft.SecurityInsights | c682a3e7-7162-4a65-bdbf-98b54a6374b1 | 8ecf8077-cf51-4820-aadd-14040956f35d | 2021-11-29 11:20:14.994000+00:00 | Azure | None | None | None | AzureActivity | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourcegroups/soc/providers/microsoft.opera... | ||||
| 1 | Check user authorization and license | Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action | Informational | Started | Started | soc | d1d8779d-38d7-4f06-91db-9cbc8de0176f | 2aaa092a-7b3c-4a77-a056-57ae1a59f7aa | Markus.Falkowski@microsoft.com | 176.199.184.128 | Administrative | Administrative | {\r\n "clientRequestId": "8bdab8e4-50bb-4108-a814-1baa55ebe019",\r\n "clientIpAddress": "176.1... | {\r\n "eventCategory": "Administrative",\r\n "entity": "/subscriptions/d1d8779d-38d7-4f06-91db... | 2021-11-29 11:21:33.173000+00:00 | {\r\n "action": "Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action",\r\n "scop... | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourceGroups/soc/providers/Microsoft.Opera... | 945c61d7-e243-4ced-b5b8-66e1e93f7369 | Microsoft.SecurityInsights | Microsoft.SecurityInsights | Microsoft.SecurityInsights | 529832af-de9a-40d9-b4d1-a82a5da34302 | 8ecf8077-cf51-4820-aadd-14040956f35d | 2021-11-29 11:20:14.995000+00:00 | Azure | None | None | None | AzureActivity | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourcegroups/soc/providers/microsoft.opera... | ||||
| 2 | Check user authorization and license | Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action | Informational | Started | Started | soc | d1d8779d-38d7-4f06-91db-9cbc8de0176f | 2aaa092a-7b3c-4a77-a056-57ae1a59f7aa | Markus.Falkowski@microsoft.com | 176.199.184.128 | Administrative | Administrative | {\r\n "clientRequestId": "8bdab8e4-50bb-4108-a814-1baa55ebe019",\r\n "clientIpAddress": "176.1... | {\r\n "eventCategory": "Administrative",\r\n "entity": "/subscriptions/d1d8779d-38d7-4f06-91db... | 2021-11-29 11:21:39.153000+00:00 | {\r\n "action": "Microsoft.SecurityInsights/dataConnectorsCheckRequirements/action",\r\n "scop... | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourceGroups/soc/providers/Microsoft.Opera... | 9cb7dbe5-e3f7-4dad-95e8-b46da3d4ac72 | Microsoft.SecurityInsights | Microsoft.SecurityInsights | Microsoft.SecurityInsights | fcbe3a89-a8d2-4796-aa38-70dfa32461f8 | 8ecf8077-cf51-4820-aadd-14040956f35d | 2021-11-29 11:20:15.001000+00:00 | Azure | None | None | None | AzureActivity | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de0176f/resourcegroups/soc/providers/microsoft.opera... |
add_query_items parameter#
Add arbitrary query statements
qry_prov_az.Azure.list_azure_activity_for_ip(
ip_address_list="176.199.184.128",
add_query_items="| summarize count() by OperationName"
)
| OperationName | count_ | |
|---|---|---|
| 0 | 360 | |
| 1 | Check user authorization and license | 181 |
| 2 | Microsoft.SecurityInsights/Incidents/investigations/write | 12 |
| 3 | Gets workflow recommend operation groups | 46 |
Splitting queries#
Divide queries into equal time chunks. Useful for queries that exceed timeouts or data limits
Caveats:
Only works with template queries
Won’t work correctly with joins in some cases
Won’t work correctly with aggregates/summaries
You must supply
startandendparameters explicitly (datetime values)
from datetime import timedelta
df = qry_prov_az.Azure.list_azure_activity_for_ip(
ip_address_list="176.199.184.128",
start=datetime.now() - timedelta(days=1),
end=datetime.utcnow(),
split_query_by="6H",
)
df[["OperationName", "TimeGenerated"]].groupby("OperationName").count()
Running: 100%|██████████| 5/5 [00:09<00:00, 1.84s/sub-queries]
| TimeGenerated | |
|---|---|
| OperationName | |
| 276 | |
| Check user authorization and license | 140 |
| Gets workflow recommend operation groups | 46 |
| Microsoft.SecurityInsights/Incidents/investigations/write | 8 |
Creating Custom queries - Basic format#
YAML files
Kusto/Log Analytics example#
metadata:
version: 1
description: Linux Syslog Host Activity Queries
data_environments: [LogAnalytics]
data_families: [LinuxSyslog]
tags: ['linux', 'syslog']
defaults:
metadata:
data_source: 'linux_syslog'
pivot:
direct_func_entities:
- Host
parameters:
table:
description: Table name
type: str
default: 'Syslog'
start:
description: Query start time
type: datetime
end:
description: Query end time
type: datetime
add_query_items:
description: Additional query clauses
type: str
default: ''
sources:
all_syslog:
description: Returns all syslog activity for a host
args:
query: '
{table}
| where TimeGenerated >= datetime({start})
| where TimeGenerated <= datetime({end})
| where Computer has "{host_name}"
{add_query_items}'
parameters:
host_name:
description: Host name or FQDN
type: str
user_group_activity:
description: All user/group additions, deletions, and modifications
args:
...
Splunk example#
list_all_alerts:
description: Retrieves all configured alerts
metadata:
data_families: [Alerts]
args:
query: '
| rest/servicesNS/-/search/saved/searches
| search alert.track=1
| fields title description search disabled triggered_alert_count actions action.script.filename alert.severity cron_schedule'
parameters:
OData Example#
list_alerts_for_user:
description: Retrieves list of alerts for a user account
metadata:
data_source: 'graph_alert'
args:
query: '{path}?$filter=createdDateTime ge {start}
and createdDateTime le {end}
and (userStates/any(d:tolower(d/userPrincipalName) eq tolower("{user_principal_name}")
or userStates/any(d:tolower(d/accountName) eq tolower("{account_name}"))
{add_query_items}'
uri: None
Specifying custom path for queries#
At runtime or in config
qry_prov = QueryProvider("M365", query_paths=["./myqueries"])
msticpyconfig.yaml#
QueryDefinitions:
Custom:
- ~/my_queries
- ~/my_other_queries
Enrichment - Threat intelligence#
Note: These sections will use a number of live services, you may not have credentials for these services so they may not run correctly for you.
A very common element of security investigations is to enrich data with Threat Intelligence data from external parties. To support his MSTICPy has integrations with a number of Threat Intelligence providers including:
VirusTotal
IBM XForce
RiskIQ
AlienVault OTX
GreyNoise
Note: Many of these integrations require an authentication key to be able to use. You can specify these in your msticpyconfig file.
Threat Intelligence integrations can be accessed using the TILookup class or via pivot providers. In the demo below we will use both methods.
# First we create our provider
ti_lookup = TILookup()
# Then we lookup results
ti_results = ti_lookup.lookup_ioc("91.211.89.33")
# Convert results to a DataFrame for ease of viewing
ti_results = ti_lookup.result_to_df(ti_results)
ti_results
Using Open PageRank. See https://www.domcop.com/openpagerank/what-is-openpagerank
| Ioc | IocType | QuerySubtype | Provider | Result | Severity | Details | RawResult | Reference | Status | |
|---|---|---|---|---|---|---|---|---|---|---|
| OTX | 91.211.89.33 | ipv4 | None | OTX | True | high | {'pulse_count': 3, 'names': ['Public report on attacks in Middle East we attribute to WIRTE APT'... | {'whois': 'http://whois.domaintools.com/91.211.89.33', 'reputation': 0, 'indicator': '91.211.89.... | https://otx.alienvault.com/api/v1/indicators/IPv4/91.211.89.33/general | 0 |
| OPR | 91.211.89.33 | ipv4 | None | OPR | False | information | IoC type ipv4 not supported. | None | None | 1 |
| RiskIQ | 91.211.89.33 | ipv4 | None | RiskIQ | False | unknown | ERROR: Error #402 "Reputation score not included in license level" (https://api.passivetotal.org... | Error #402 "Reputation score not included in license level" (https://api.passivetotal.org/v2/rep... | https://community.riskiq.com | 3 |
| Tor | 91.211.89.33 | ipv4 | None | Tor | True | information | Not found. | None | https://check.torproject.org/exit-addresses | 0 |
| VirusTotal | 91.211.89.33 | ipv4 | None | VirusTotal | True | information | {'verbose_msg': 'IP address in dataset', 'response_code': 1, 'positives': 0, 'detected_urls': []} | {'asn': 206638, 'undetected_urls': [['http://91.211.89.33/', '534745024fb0a0b687cf79d0f037042a07... | https://www.virustotal.com/vtapi/v2/ip-address/report | 0 |
| XForce | 91.211.89.33 | ipv4 | None | XForce | True | information | {'score': 1, 'cats': {}, 'categoryDescriptions': {}, 'reason': 'Regional Internet Registry', 're... | {'ip': '91.211.89.33', 'history': [{'created': '2012-03-22T07:26:00.000Z', 'reason': 'Regional I... | https://api.xforce.ibmcloud.com/ipr/91.211.89.33 | 0 |
# We can also display them in a browser
TILookup.browse_results(ti_results)
91.211.89.33
Type: 'ipv4', Provider: OTX, severity: high
Details
| OTX | |
| pulse_count | 3 |
| names | ['Public report on attacks in Middle East we attribute to WIRTE APT', 'WIRTE’s campaign in the Middle East ‘living off the land’ since at least 2019', 'test2'] |
| tags | [['wirte', 'macros', 'microsoft excel', 'spear phishing'], ['hkcu', 'class ids', 'appdata', 'programdata'], []] |
| references | [['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'], ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'], []] |
Reference:
https://otx.alienvault.com/api/v1/indicators/IPv4/91.211.89.33/generalRaw Results
Raw results from provider...
{'accuracy_radius': 500,
'area_code': 0,
'asn': 'AS206638 PE Brezhnev Daniil',
'base_indicator': {'access_reason': '',
'access_type': 'public',
'content': '',
'description': '',
'id': 2822432629,
'indicator': '91.211.89.33',
'title': '',
'type': 'IPv4'},
'charset': 0,
'city': 'Dnipro',
'city_data': True,
'continent_code': 'EU',
'country_code': 'UA',
'country_code2': 'UA',
'country_code3': 'UKR',
'country_name': 'Ukraine',
'dma_code': 0,
'false_positive': [],
'flag_title': 'Ukraine',
'flag_url': '/assets/images/flags/ua.png',
'indicator': '91.211.89.33',
'latitude': 48.4735,
'longitude': 35.046,
'postal_code': '49028',
'pulse_info': {'count': 3,
'pulses': [{'TLP': 'white',
'adversary': 'WIRTE',
'attack_ids': [{'display_name': 'T1059 - Command '
'and Scripting '
'Interpreter',
'id': 'T1059',
'name': 'Command and Scripting '
'Interpreter'},
{'display_name': 'T1218 - Signed '
'Binary Proxy '
'Execution',
'id': 'T1218',
'name': 'Signed Binary Proxy '
'Execution'},
{'display_name': 'T1113 - Screen '
'Capture',
'id': 'T1113',
'name': 'Screen Capture'},
{'display_name': 'T1546 - Event '
'Triggered '
'Execution',
'id': 'T1546',
'name': 'Event Triggered '
'Execution'},
{'display_name': 'T1562 - Impair '
'Defenses',
'id': 'T1562',
'name': 'Impair Defenses'},
{'display_name': 'T1036 - '
'Masquerading',
'id': 'T1036',
'name': 'Masquerading'},
{'display_name': 'T1566 - Phishing',
'id': 'T1566',
'name': 'Phishing'},
{'display_name': 'T1137.001 - '
'Office Template '
'Macros',
'id': 'T1137.001',
'name': 'Office Template Macros'},
{'display_name': 'T1059.001 - '
'PowerShell',
'id': 'T1059.001',
'name': 'PowerShell'},
{'display_name': 'T1193 - '
'Spearphishing '
'Attachment',
'id': 'T1193',
'name': 'Spearphishing '
'Attachment'}],
'author': {'avatar_url': '/otxapi/users/avatar_image/media/avatars/user_2/resized/80/avatar_dacfad0ca8.png',
'id': '2',
'is_following': False,
'is_subscribed': True,
'username': 'AlienVault'},
'cloned_from': None,
'comment_count': 0,
'created': '2021-11-29T16:10:35.042000',
'description': 'This February researchers came '
'across MS Excel droppers that use '
'hidden spreadsheets and VBA macros '
'to drop their first stage implant. '
'The implant itself is a VBS script '
'with functionality to collect '
'system information and execute '
'arbitrary code sent by the '
'attackers on the infected machine. '
'Although these intrusion sets may '
'appear similar to the new '
'MuddyWater first stage VBS implant '
'used for reconnaissance and '
'profiling activities, they have '
'slightly different TTPs and wider '
'targeting. To date, most of the '
'known victims are located in the '
'Middle East, but there are also '
'targets in other regions',
'downvotes_count': 0,
'export_count': 27,
'follower_count': 0,
'groups': [],
'id': '61a4fb7c9b88f16b103c151d',
'in_group': False,
'indicator_count': 26,
'indicator_type_counts': {'FileHash-MD5': 6,
'FileHash-SHA1': 2,
'FileHash-SHA256': 2,
'IPv4': 4,
'domain': 12},
'industries': ['Telecommunications',
'Energy',
'Political',
'Technology',
'Military',
'Financial',
'Diplomatic',
'Government'],
'is_author': False,
'is_modified': False,
'is_subscribing': None,
'locked': False,
'malware_families': [],
'modified': '2021-11-29T16:10:35.042000',
'modified_text': '10 hours ago ',
'name': 'Public report on attacks in Middle East '
'we attribute to WIRTE APT',
'public': 1,
'pulse_source': 'web',
'references': ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'],
'related_indicator_is_active': 1,
'related_indicator_type': 'IPv4',
'subscriber_count': 166586,
'tags': ['wirte',
'macros',
'microsoft excel',
'spear phishing'],
'targeted_countries': [],
'threat_hunter_has_agents': 1,
'threat_hunter_scannable': True,
'upvotes_count': 0,
'validator_count': 0,
'vote': 0,
'votes_count': 0},
{'TLP': 'white',
'adversary': '',
'attack_ids': [],
'author': {'avatar_url': '/otxapi/users/avatar_image/media/avatars/user_94093/resized/80/avatar_281f69b768.png',
'id': '94093',
'is_following': False,
'is_subscribed': False,
'username': 'Sand-Storm'},
'cloned_from': None,
'comment_count': 0,
'created': '2021-11-29T17:27:20.561000',
'description': 'Malicious documents and droppers, '
'as reported by the BBC, have been '
'uncovered by researchers at the '
'University of Kansas in the United '
'States and are subject to an '
'investigation by security firm '
'Kaspersky.',
'downvotes_count': 0,
'export_count': 2,
'follower_count': 0,
'groups': [],
'id': '61a50d7881106990abd40bd1',
'in_group': False,
'indicator_count': 23,
'indicator_type_counts': {'FileHash-MD5': 6,
'FileHash-SHA1': 2,
'FileHash-SHA256': 2,
'IPv4': 4,
'domain': 9},
'industries': [],
'is_author': False,
'is_modified': False,
'is_subscribing': None,
'locked': False,
'malware_families': [],
'modified': '2021-11-29T17:27:20.561000',
'modified_text': '9 hours ago ',
'name': 'WIRTE’s campaign in the Middle East '
'‘living off the land’ since at least 2019',
'public': 1,
'pulse_source': 'web',
'references': ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'],
'related_indicator_is_active': 1,
'related_indicator_type': 'IPv4',
'subscriber_count': 275,
'tags': ['hkcu',
'class ids',
'appdata',
'programdata'],
'targeted_countries': [],
'threat_hunter_has_agents': 1,
'threat_hunter_scannable': True,
'upvotes_count': 0,
'validator_count': 0,
'vote': 0,
'votes_count': 0},
{'TLP': 'green',
'adversary': '',
'attack_ids': [],
'author': {'avatar_url': 'https://otx.alienvault.com/assets/images/default-avatar.png',
'id': '157433',
'is_following': False,
'is_subscribed': False,
'username': 'SURMI003'},
'cloned_from': None,
'comment_count': 0,
'created': '2021-08-03T04:46:56.557000',
'description': "The world's largest body of human "
'evolution has a new leader - and '
"it's got a lot of work to do - to "
"make sure it doesn't look like it "
'is going to get any worse.',
'downvotes_count': 0,
'export_count': 0,
'follower_count': 0,
'groups': [],
'id': '6108ca40db1c7b5f1986e2de',
'in_group': False,
'indicator_count': 0,
'indicator_type_counts': {},
'industries': [],
'is_author': False,
'is_modified': True,
'is_subscribing': None,
'locked': False,
'malware_families': [],
'modified': '2021-09-02T00:01:54.965000',
'modified_text': '89 days ago ',
'name': 'test2',
'public': 1,
'pulse_source': 'web',
'references': [],
'related_indicator_is_active': 0,
'related_indicator_type': 'IPv4',
'subscriber_count': 11,
'tags': [],
'targeted_countries': [],
'threat_hunter_has_agents': 1,
'threat_hunter_scannable': False,
'upvotes_count': 0,
'validator_count': 0,
'vote': 0,
'votes_count': 0}],
'references': ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'],
'related': {'alienvault': {'adversary': ['WIRTE'],
'industries': ['Energy',
'Financial',
'Technology',
'Political',
'Telecommunications',
'Government',
'Military',
'Diplomatic'],
'malware_families': []},
'other': {'adversary': [],
'industries': [],
'malware_families': []}}},
'region': '12',
'reputation': 0,
'sections': ['general',
'geo',
'reputation',
'url_list',
'passive_dns',
'malware',
'nids_list',
'http_scans'],
'subdivision': '12',
'type': 'IPv4',
'type_title': 'IPv4',
'validation': [],
'whois': 'http://whois.domaintools.com/91.211.89.33'}
We can do the exact same set of lookups directly from that IP address if we define it as an entity:
from msticpy.datamodel.entities import IpAddress
pivot = Pivot(namespace=globals())
destip = IpAddress(Address="91.211.89.33")
ti_results = destip.tilookup_ipv4()
TILookup.browse_results(ti_results)
This product includes GeoLite2 data created by MaxMind, available from
https://www.maxmind.com.
This library uses services provided by ipstack.
https://ipstack.com
91.211.89.33
Type: 'ipv4', Provider: OTX, severity: high
Details
| OTX | |
| pulse_count | 3 |
| names | ['Public report on attacks in Middle East we attribute to WIRTE APT', 'WIRTE’s campaign in the Middle East ‘living off the land’ since at least 2019', 'test2'] |
| tags | [['wirte', 'macros', 'microsoft excel', 'spear phishing'], ['hkcu', 'class ids', 'appdata', 'programdata'], []] |
| references | [['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'], ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'], []] |
Reference:
https://otx.alienvault.com/api/v1/indicators/IPv4/91.211.89.33/generalRaw Results
Raw results from provider...
{'accuracy_radius': 500,
'area_code': 0,
'asn': 'AS206638 PE Brezhnev Daniil',
'base_indicator': {'access_reason': '',
'access_type': 'public',
'content': '',
'description': '',
'id': 2822432629,
'indicator': '91.211.89.33',
'title': '',
'type': 'IPv4'},
'charset': 0,
'city': 'Dnipro',
'city_data': True,
'continent_code': 'EU',
'country_code': 'UA',
'country_code2': 'UA',
'country_code3': 'UKR',
'country_name': 'Ukraine',
'dma_code': 0,
'false_positive': [],
'flag_title': 'Ukraine',
'flag_url': '/assets/images/flags/ua.png',
'indicator': '91.211.89.33',
'latitude': 48.4735,
'longitude': 35.046,
'postal_code': '49028',
'pulse_info': {'count': 3,
'pulses': [{'TLP': 'white',
'adversary': 'WIRTE',
'attack_ids': [{'display_name': 'T1059 - Command '
'and Scripting '
'Interpreter',
'id': 'T1059',
'name': 'Command and Scripting '
'Interpreter'},
{'display_name': 'T1218 - Signed '
'Binary Proxy '
'Execution',
'id': 'T1218',
'name': 'Signed Binary Proxy '
'Execution'},
{'display_name': 'T1113 - Screen '
'Capture',
'id': 'T1113',
'name': 'Screen Capture'},
{'display_name': 'T1546 - Event '
'Triggered '
'Execution',
'id': 'T1546',
'name': 'Event Triggered '
'Execution'},
{'display_name': 'T1562 - Impair '
'Defenses',
'id': 'T1562',
'name': 'Impair Defenses'},
{'display_name': 'T1036 - '
'Masquerading',
'id': 'T1036',
'name': 'Masquerading'},
{'display_name': 'T1566 - Phishing',
'id': 'T1566',
'name': 'Phishing'},
{'display_name': 'T1137.001 - '
'Office Template '
'Macros',
'id': 'T1137.001',
'name': 'Office Template Macros'},
{'display_name': 'T1059.001 - '
'PowerShell',
'id': 'T1059.001',
'name': 'PowerShell'},
{'display_name': 'T1193 - '
'Spearphishing '
'Attachment',
'id': 'T1193',
'name': 'Spearphishing '
'Attachment'}],
'author': {'avatar_url': '/otxapi/users/avatar_image/media/avatars/user_2/resized/80/avatar_dacfad0ca8.png',
'id': '2',
'is_following': False,
'is_subscribed': True,
'username': 'AlienVault'},
'cloned_from': None,
'comment_count': 0,
'created': '2021-11-29T16:10:35.042000',
'description': 'This February researchers came '
'across MS Excel droppers that use '
'hidden spreadsheets and VBA macros '
'to drop their first stage implant. '
'The implant itself is a VBS script '
'with functionality to collect '
'system information and execute '
'arbitrary code sent by the '
'attackers on the infected machine. '
'Although these intrusion sets may '
'appear similar to the new '
'MuddyWater first stage VBS implant '
'used for reconnaissance and '
'profiling activities, they have '
'slightly different TTPs and wider '
'targeting. To date, most of the '
'known victims are located in the '
'Middle East, but there are also '
'targets in other regions',
'downvotes_count': 0,
'export_count': 27,
'follower_count': 0,
'groups': [],
'id': '61a4fb7c9b88f16b103c151d',
'in_group': False,
'indicator_count': 26,
'indicator_type_counts': {'FileHash-MD5': 6,
'FileHash-SHA1': 2,
'FileHash-SHA256': 2,
'IPv4': 4,
'domain': 12},
'industries': ['Telecommunications',
'Energy',
'Political',
'Technology',
'Military',
'Financial',
'Diplomatic',
'Government'],
'is_author': False,
'is_modified': False,
'is_subscribing': None,
'locked': False,
'malware_families': [],
'modified': '2021-11-29T16:10:35.042000',
'modified_text': '10 hours ago ',
'name': 'Public report on attacks in Middle East '
'we attribute to WIRTE APT',
'public': 1,
'pulse_source': 'web',
'references': ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'],
'related_indicator_is_active': 1,
'related_indicator_type': 'IPv4',
'subscriber_count': 166586,
'tags': ['wirte',
'macros',
'microsoft excel',
'spear phishing'],
'targeted_countries': [],
'threat_hunter_has_agents': 1,
'threat_hunter_scannable': True,
'upvotes_count': 0,
'validator_count': 0,
'vote': 0,
'votes_count': 0},
{'TLP': 'white',
'adversary': '',
'attack_ids': [],
'author': {'avatar_url': '/otxapi/users/avatar_image/media/avatars/user_94093/resized/80/avatar_281f69b768.png',
'id': '94093',
'is_following': False,
'is_subscribed': False,
'username': 'Sand-Storm'},
'cloned_from': None,
'comment_count': 0,
'created': '2021-11-29T17:27:20.561000',
'description': 'Malicious documents and droppers, '
'as reported by the BBC, have been '
'uncovered by researchers at the '
'University of Kansas in the United '
'States and are subject to an '
'investigation by security firm '
'Kaspersky.',
'downvotes_count': 0,
'export_count': 2,
'follower_count': 0,
'groups': [],
'id': '61a50d7881106990abd40bd1',
'in_group': False,
'indicator_count': 23,
'indicator_type_counts': {'FileHash-MD5': 6,
'FileHash-SHA1': 2,
'FileHash-SHA256': 2,
'IPv4': 4,
'domain': 9},
'industries': [],
'is_author': False,
'is_modified': False,
'is_subscribing': None,
'locked': False,
'malware_families': [],
'modified': '2021-11-29T17:27:20.561000',
'modified_text': '9 hours ago ',
'name': 'WIRTE’s campaign in the Middle East '
'‘living off the land’ since at least 2019',
'public': 1,
'pulse_source': 'web',
'references': ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'],
'related_indicator_is_active': 1,
'related_indicator_type': 'IPv4',
'subscriber_count': 275,
'tags': ['hkcu',
'class ids',
'appdata',
'programdata'],
'targeted_countries': [],
'threat_hunter_has_agents': 1,
'threat_hunter_scannable': True,
'upvotes_count': 0,
'validator_count': 0,
'vote': 0,
'votes_count': 0},
{'TLP': 'green',
'adversary': '',
'attack_ids': [],
'author': {'avatar_url': 'https://otx.alienvault.com/assets/images/default-avatar.png',
'id': '157433',
'is_following': False,
'is_subscribed': False,
'username': 'SURMI003'},
'cloned_from': None,
'comment_count': 0,
'created': '2021-08-03T04:46:56.557000',
'description': "The world's largest body of human "
'evolution has a new leader - and '
"it's got a lot of work to do - to "
"make sure it doesn't look like it "
'is going to get any worse.',
'downvotes_count': 0,
'export_count': 0,
'follower_count': 0,
'groups': [],
'id': '6108ca40db1c7b5f1986e2de',
'in_group': False,
'indicator_count': 0,
'indicator_type_counts': {},
'industries': [],
'is_author': False,
'is_modified': True,
'is_subscribing': None,
'locked': False,
'malware_families': [],
'modified': '2021-09-02T00:01:54.965000',
'modified_text': '89 days ago ',
'name': 'test2',
'public': 1,
'pulse_source': 'web',
'references': [],
'related_indicator_is_active': 0,
'related_indicator_type': 'IPv4',
'subscriber_count': 11,
'tags': [],
'targeted_countries': [],
'threat_hunter_has_agents': 1,
'threat_hunter_scannable': False,
'upvotes_count': 0,
'validator_count': 0,
'vote': 0,
'votes_count': 0}],
'references': ['https://securelist.com/wirtes-campaign-in-the-middle-east-living-off-the-land-since-at-least-2019/105044/'],
'related': {'alienvault': {'adversary': ['WIRTE'],
'industries': ['Energy',
'Financial',
'Technology',
'Political',
'Telecommunications',
'Government',
'Military',
'Diplomatic'],
'malware_families': []},
'other': {'adversary': [],
'industries': [],
'malware_families': []}}},
'region': '12',
'reputation': 0,
'sections': ['general',
'geo',
'reputation',
'url_list',
'passive_dns',
'malware',
'nids_list',
'http_scans'],
'subdivision': '12',
'type': 'IPv4',
'type_title': 'IPv4',
'validation': [],
'whois': 'http://whois.domaintools.com/91.211.89.33'}
Its also possible to look up multiple IoCs at once from a DataFrame:
import pandas as pd
host_logons = pd.read_pickle("../data/host_logons.pkl")
ti_df = ti_lookup.lookup_iocs(host_logons, obs_col = "IpAddress")
ti_df.head()
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_52424/1394424922.py in <module>
1 import pandas as pd
----> 2 host_logons = pd.read_pickle("data/host_logons.pkl")
3 ti_df = ti_lookup.lookup_iocs(host_logons, obs_col = "IpAddress")
4 ti_df.head()
~\AppData\Roaming\Python\Python37\site-packages\pandas\io\pickle.py in read_pickle(filepath_or_buffer, compression, storage_options)
176 compression=compression,
177 is_text=False,
--> 178 storage_options=storage_options,
179 ) as handles:
180
~\AppData\Roaming\Python\Python37\site-packages\pandas\io\common.py in get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options)
649 else:
650 # Binary mode
--> 651 handle = open(handle, ioargs.mode)
652 handles.append(handle)
653
FileNotFoundError: [Errno 2] No such file or directory: 'data/host_logons.pkl'
IpAddress.tilookup_ipv4(host_logons, input_column="IpAddress")
Its not just IP addresses either, there is support for a wide range of entity types:
from msticpy.datamodel.entities import *
md("Domains name:", "bold")
display(Dns.tilookup_dns("energyxprt.com"))
md("Files:", "bold")
display(File.tilookup_file_hash("f11750939680d28d724a73bb0830a04fa7b926aead104ad7d0d8d76df634686f"))
md("Urls:", "bold")
display(Url.tilookup_url("http://saounps.com/netfV2/932ec2a5c5a97f160704d4c6e5d32a93"))
More details on Threat Intelligence lookups in MSTICPy can be found in the documenation: https://msticpy.readthedocs.io/en/latest/data_acquisition/TIProviders.html
Enrichment - Other enrichment [Pete]#
Another key feature of MSTICPy is the ability to enrich your core security log data with additional data sources that help provide additional information and context to a security analyst.
There are a number of data enrichments avaliable including:
GeoIP data to locate an IP Address
WhoIs data to provide information on a domain owner
Azure API data to provide additional data on Azure resources.
IP Tools#
MSTICPy contains a number of IP related enrichments the are grouped under the IPAddress entity type.
GeoIP is a useful feature to enrich your data with information about the location of an IP address and provide context about whether the IP address shoudl be considered suspicious or not.
MSTICPy supports getting GeoIP data from tow key sources, MaxMind’s GeoIPLite service, and the IPStack Geo service.
Note: Both of these GeoIP services require an API key - more details can be found in the MSTICPy documentation
from msticpy.datamodel.pivot import Pivot
Pivot(namespace=globals())
from msticpy.datamodel.entities import IpAddress
IpAddress.util.geoloc(value="103.125.190.248")
MSTICPy also has IP tools to get WhoIs information on an IP address:
IpAddress.util.whois(value="103.125.190.248")
And to do a resverse DNS lookup:
IpAddress.util.ip_rev_resolve(value="103.125.190.248")
Domain Tools#
Similar enrichments exist for other common entity types such as domains (under the Dns entity type):
from msticpy.datamodel.entities import Dns
Dns.util.dns_resolve("www.contoso.com")
Dns.util.dns_in_abuse_list("www.contoso.com")
We can also fetch a screenshot of a target URL in order to give the analyst a visual representation of the site being investigated.
Note: Screenshots are enabled by the Browshot service
from msticpy.sectools.domain_utils import screenshot
from IPython.display import display, Image
sshot = screenshot("www.contoso.com")
with open('screenshot.png', 'wb') as f:
f.write(sshot.content)
display(Image(filename='screenshot.png'))
from msticpy.sectools.geoip import GeoLiteLookup
iplocation = GeoLiteLookup()
loc_result, ip_entity = iplocation.lookup_ip(ip_address='90.156.201.97')
print('Raw result')
display(loc_result)
print('IP Address Entity')
display(ip_entity[0])
We can also lookup multiple IP addresses at once by passing in a list of IP addresses.
ips = ["103.125.190.248", "173.232.207.214", "52.200.40.111"]
_, ip_entities = iplocation.lookup_ip(ip_addr_list=ips)
ents = [ip_ent.properties for ip_ent in ip_entities]
pd.DataFrame(ents)
MSTICPy also has tools to support getting information such as the WhoIs record for an IP address:
from msticpy.sectools.ip_utils import get_whois_info
get_whois_info("103.125.190.248")
Azure Data Enrichment#
MSTICPy also includes a number of Azure API integrations that can be used to enrich your data with additional data about Azure Resources. These are avaliable in two fromats, via the AzureData feature of MSTICPy and also via the new Azure Resource Graph data connector.
from msticpy.data.azure_data import AzureData
# Create our Azure Data instance and connect
az_data = AzureData()
az_data.connect()
# List subscriptions our account has access to
subs = az_data.get_subscriptions()
subs.head()
Attempting to sign-in with environment variable credentials...
Attempting to sign-in with environment variable credentials...
| Subscription ID | Display Name | State | |
|---|---|---|---|
| 0 | 8c4b5b03-3b24-4ed0-91f5-a703cd91b412 | Cosmos_C&E_Azure_AzureEngineeringSystems_100200 | Enabled |
| 1 | 7a7b5559-58af-401a-b543-61b7321a97ea | Epic-Edge-ES-GitCache-Prod | Enabled |
| 2 | bac420ed-c6fc-4a05-8ac1-8c0c52da1d6e | IDEAs MS Reporting | Enabled |
| 3 | 7fd08dcc-a653-4b0f-8f8c-4dac889fdda4 | Code generate Test and Infra | Enabled |
| 4 | 54b875cc-a81a-4914-8bfd-1a36bc7ddf4d | MSFT-WindowsVirtualDesktop-01 | Enabled |
subscription = subs[subs['Display Name'].str.contains("ASI Hunting Demo")].iloc[0]
subscription
Subscription ID 40dcc8bf-0478-4f3b-b275-ed0a94f2c013
Display Name ASI Hunting Demo Environment
State Enabled
Name: 84, dtype: object
# List resources in our subscription
sub_info = az_data.get_subscription_info(sub_id=subscription['Subscription ID'])
display(sub_info)
{'Subscription ID': '40dcc8bf-0478-4f3b-b275-ed0a94f2c013',
'Display Name': 'ASI Hunting Demo Environment',
'State': 'Enabled',
'Subscription Location': 'Internal_2014-09-01',
'Subscription Quota': 'Internal_2014-09-01',
'Spending Limit': 'Off'}
We can also use the AzureSentinel feature to get details about specific Microsoft Sentitnel elements:
from msticpy.data.azure_sentinel import AzureSentinel
azure_sent = AzureSentinel()
azure_sent.connect()
Attempting to sign-in with environment variable credentials...
Attempting to sign-in with environment variable credentials...
# List our workspaces
azure_sent.get_sentinel_workspaces(sub_id=subscription['Subscription ID'])
Finding Azure Sentinel Workspaces...
Attempting to sign-in with environment variable credentials...
{'TempLAWorkspace': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/test/providers/Microsoft.OperationalInsights/workspaces/TempLAWorkspace',
'koreasentinelworkspace': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/korearg/providers/Microsoft.OperationalInsights/workspaces/koreasentinelworkspace',
'franceworkspace': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/france-rg/providers/Microsoft.OperationalInsights/workspaces/franceworkspace',
'ASIHuntOMSWorkspaceV4': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/ASIHuntOMSWorkspaceRG/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4',
'westeuroworkspace': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/westeuro-rg/providers/Microsoft.OperationalInsights/workspaces/westeuroworkspace',
'austeastwkspc': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/ASIWorkspaceRG/providers/Microsoft.OperationalInsights/workspaces/austeastwkspc',
'sentinellayounes': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/splunkrg/providers/Microsoft.OperationalInsights/workspaces/sentinellayounes',
'weusworkspace': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/westusrg/providers/Microsoft.OperationalInsights/workspaces/weusworkspace',
'weusworkspace2': '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/westusrg/providers/Microsoft.OperationalInsights/workspaces/weusworkspace2'}
# Get incidents from the workspace
incidents = azure_sent.get_incidents(res_id = '/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourcegroups/ASIHuntOMSWorkspaceRG/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4')
incidents.head()
| id | name | etag | type | properties.title | properties.description | properties.severity | properties.status | properties.owner.objectId | properties.owner.email | properties.owner.assignedTo | properties.owner.userPrincipalName | properties.labels | properties.firstActivityTimeUtc | properties.lastActivityTimeUtc | properties.lastModifiedTimeUtc | properties.createdTimeUtc | properties.incidentNumber | properties.additionalData.alertsCount | properties.additionalData.bookmarksCount | properties.additionalData.commentsCount | properties.additionalData.alertProductNames | properties.additionalData.tactics | properties.relatedAnalyticRuleIds | properties.incidentUrl | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | /subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provide... | 1a070aec-12a2-49f0-ab9d-cb712b49d49a | "02003475-0000-0a00-0000-618b30f10000" | Microsoft.SecurityInsights/Incidents | Unauthenticated access to a storage blob container | Unusual unauthenticated access to your storage account 'asicsvstorage' was detected. Access to t... | Medium | New | None | None | None | None | [] | 2021-11-10T01:43:11.41Z | 2021-11-10T01:43:11.41Z | 2021-11-10T02:39:45.3489835Z | 2021-11-10T02:39:45.3489835Z | 46150 | 1 | 0 | 0 | [Azure Security Center] | [InitialAccess] | [/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provid... | https://portal.azure.com/#asset/Microsoft_Azure_Security_Insights/Incident/subscriptions/40dcc8b... |
| 1 | /subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provide... | f4f87d6f-1755-4bf2-9e82-e929d6446bf6 | "b2014a58-0000-0a00-0000-615f79010000" | Microsoft.SecurityInsights/Incidents | Unauthenticated access to a storage blob container | Unusual unauthenticated access to your storage account 'asicsvstorage' was detected. Access to t... | Medium | New | None | None | None | None | [] | 2021-10-07T21:00:59.79Z | 2021-10-07T21:00:59.79Z | 2021-10-07T22:47:29.2329894Z | 2021-10-07T22:47:29.2329894Z | 46149 | 1 | 0 | 0 | [Azure Security Center] | [InitialAccess] | [/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provid... | https://portal.azure.com/#asset/Microsoft_Azure_Security_Insights/Incident/subscriptions/40dcc8b... |
| 2 | /subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provide... | 24c8e377-c76a-4ee0-bb17-720768742fc2 | "a201a7a1-0000-0a00-0000-615e17430000" | Microsoft.SecurityInsights/Incidents | PREVIEW - Suspicious management session using Azure portal detected | Analysis of your subscription activity logs has detected a suspicious behavior.\nA principal tha... | Medium | New | None | None | None | None | [] | 2021-10-06T20:14:31.0557037Z | 2021-10-06T20:40:13.8362148Z | 2021-10-06T21:38:11.2303988Z | 2021-10-06T21:38:11.2303988Z | 46148 | 1 | 0 | 0 | [Azure Security Center] | [Persistence] | [/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provid... | https://portal.azure.com/#asset/Microsoft_Azure_Security_Insights/Incident/subscriptions/40dcc8b... |
| 3 | /subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provide... | f7b36bc7-aca6-4327-8e6a-a45671c0d316 | "d800ed95-0000-0a00-0000-612554db0000" | Microsoft.SecurityInsights/Incidents | Unusual application accessed a storage file share | Someone has accessed your Azure storage account 'ianhellepub2020382608650' using an unexpected a... | Medium | New | None | None | None | None | [] | 2021-08-24T18:46:03Z | 2021-08-24T18:46:03Z | 2021-08-24T20:21:47.1463327Z | 2021-08-24T20:21:47.1463327Z | 46147 | 1 | 0 | 0 | [Azure Security Center] | [InitialAccess] | [/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provid... | https://portal.azure.com/#asset/Microsoft_Azure_Security_Insights/Incident/subscriptions/40dcc8b... |
| 4 | /subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provide... | eed27e1a-7794-4535-b088-7463793b1748 | "40004b48-0000-0a00-0000-60f601f30000" | Microsoft.SecurityInsights/Incidents | Access from an unusual location to a storage blob container | Someone has accessed your Azure Storage account 'ngchitempaml204478910230' from an unusual locat... | Low | New | None | None | None | None | [] | 2021-07-19T21:17:57.688Z | 2021-07-19T21:17:57.688Z | 2021-07-19T22:51:31.3884926Z | 2021-07-19T22:51:31.3884926Z | 46146 | 1 | 0 | 0 | [Azure Security Center] | [InitialAccess] | [/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/ASIHuntOMSWorkspaceRG/provid... | https://portal.azure.com/#asset/Microsoft_Azure_Security_Insights/Incident/subscriptions/40dcc8b... |
We can also get similar details from the Azure Resource Graph. This is accessed in the same way as our other Query Providers:
res_graph_qry_prov = QueryProvider("ResourceGraph")
res_graph_qry_prov.connect()
vm_df = res_graph_qry_prov.ResourceGraph.list_virtual_machines()
vm_df.head()
Attempting to sign-in with environment variable credentials...
Attempting to sign-in with environment variable credentials...
Connected
Attempting to sign-in with environment variable credentials...
| id | name | type | tenantId | kind | location | resourceGroup | subscriptionId | managedBy | sku | plan | tags | zones | extendedLocation | properties.provisioningState | properties.storageProfile.imageReference.publisher | properties.storageProfile.imageReference.exactVersion | properties.storageProfile.imageReference.version | properties.storageProfile.imageReference.sku | properties.storageProfile.imageReference.offer | properties.storageProfile.dataDisks | properties.storageProfile.osDisk.name | properties.storageProfile.osDisk.createOption | properties.storageProfile.osDisk.diskSizeGB | properties.storageProfile.osDisk.managedDisk.id | ... | identity.userAssignedIdentities./subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/AzSecPackAutoConfigRG/providers/Microsoft.ManagedIdentity/userAssignedIdentities/AzSecPackAutoConfigUA-eastus.clientId | identity.userAssignedIdentities./subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/AzSecPackAutoConfigRG/providers/Microsoft.ManagedIdentity/userAssignedIdentities/AzSecPackAutoConfigUA-westus2.principalId | identity.userAssignedIdentities./subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/AzSecPackAutoConfigRG/providers/Microsoft.ManagedIdentity/userAssignedIdentities/AzSecPackAutoConfigUA-westus2.clientId | tags.testtag | tags.Demo | tags.Track | tags.CloudPlanDate | tags.Responsible | tags.Deployment_Date | tags.function | tags.BackupGroup | tags.Deployment_ID | tags.department | tags.BizEnv | tags.type | tags.Importance | tags.Business_Service | tags.datadog_monitored | tags.SAP product | tags.SAP SID | properties.proximityPlacementGroup.id | tags.Case | tags.ISO | tags.tst_vm | tags.application | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | /subscriptions/3c1bb38c-82e3-4f8d-a115-a7110ba70d05/resourceGroups/CONTOSO77/providers/Microsoft... | zscaler-miror-cef | microsoft.compute/virtualmachines | 72f988bf-86f1-41af-91ab-2d7cd011db47 | westus | contoso77 | 3c1bb38c-82e3-4f8d-a115-a7110ba70d05 | None | NaN | NaN | None | None | Succeeded | Canonical | 18.04.201907221 | latest | 18.04-LTS | UbuntuServer | [] | zscaler-miror-cef_disk1_d8f5a258115c4db1bdc8483fc71e5d4f | FromImage | 30.0 | /subscriptions/3c1bb38c-82e3-4f8d-a115-a7110ba70d05/resourceGroups/Contoso77/providers/Microsoft... | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ||
| 1 | /subscriptions/0c8d0493-55c3-4b3f-a0b0-c8d4d2ce0343/resourceGroups/ARC-E2E-Test-Dev-EUS2EUAP/pro... | zahi-d-stage-eus2euap | microsoft.compute/virtualmachines | 72f988bf-86f1-41af-91ab-2d7cd011db47 | eastus2euap | arc-e2e-test-dev-eus2euap | 0c8d0493-55c3-4b3f-a0b0-c8d4d2ce0343 | None | NaN | NaN | None | None | Succeeded | Canonical | 18.04.202111080 | latest | 18.04-LTS | UbuntuServer | [] | zahi-d-stage-eus2euap_OsDisk_1_b1accaf8d5a148f8bdd69fc51400773c | FromImage | 30.0 | /subscriptions/0c8d0493-55c3-4b3f-a0b0-c8d4d2ce0343/resourceGroups/ARC-E2E-Test-Dev-EUS2EUAP/pro... | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ||
| 2 | /subscriptions/3c1bb38c-82e3-4f8d-a115-a7110ba70d05/resourceGroups/contoso77/providers/Microsoft... | yuval-sysmon-flow | microsoft.compute/virtualmachines | 72f988bf-86f1-41af-91ab-2d7cd011db47 | eastus2 | contoso77 | 3c1bb38c-82e3-4f8d-a115-a7110ba70d05 | None | NaN | NaN | [1] | None | Succeeded | MicrosoftWindowsServer | 20348.169.2108120020 | latest | 2022-datacenter | WindowsServer | [] | yuval-sysmon-flow_OsDisk_1_0152ae17069a4320aa594f439a2e1668 | FromImage | NaN | /subscriptions/3c1bb38c-82e3-4f8d-a115-a7110ba70d05/resourceGroups/Contoso77/providers/Microsoft... | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ||
| 3 | /subscriptions/de5fb112-5d5d-42d4-a9ea-5f3b1359c6a6/resourceGroups/YUVALNAOR-RG/providers/Micros... | yuval-sysmon-flow | microsoft.compute/virtualmachines | 72f988bf-86f1-41af-91ab-2d7cd011db47 | eastus2 | yuvalnaor-rg | de5fb112-5d5d-42d4-a9ea-5f3b1359c6a6 | None | NaN | NaN | [1] | None | Succeeded | MicrosoftWindowsServer | 17763.2114.2108051826 | latest | 2019-Datacenter | WindowsServer | [] | yuval-sysmon-flow_disk1_ae66aaeb62954dfbb69a09e21c81846a | FromImage | NaN | /subscriptions/de5fb112-5d5d-42d4-a9ea-5f3b1359c6a6/resourceGroups/yuvalnaor-rg/providers/Micros... | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ||
| 4 | /subscriptions/de5fb112-5d5d-42d4-a9ea-5f3b1359c6a6/resourceGroups/YEHUDABOLYPALOALTOAPI/provide... | yehudabolyfirewallv9api | microsoft.compute/virtualmachines | 72f988bf-86f1-41af-91ab-2d7cd011db47 | eastus2 | yehudabolypaloaltoapi | de5fb112-5d5d-42d4-a9ea-5f3b1359c6a6 | None | NaN | NaN | None | None | Succeeded | paloaltonetworks | 9.1.0 | latest | byol | vmseries1 | [] | yehudabolyfirewallv9api_OsDisk_1_943495312af044f289ad3383e72b56ed | FromImage | 60.0 | /subscriptions/de5fb112-5d5d-42d4-a9ea-5f3b1359c6a6/resourceGroups/YEHUDABOLYPALOALTOAPI/provide... | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
5 rows × 230 columns
As with other Query Providers you can run custom queries:
query = """Resources
| where type == 'microsoft.compute/virtualmachines'
| summarize count() by PowerState = tostring(properties.extended.instanceView.powerState.code)"""
res_graph_qry_prov.exec_query(query)
| PowerState | count_ | |
|---|---|---|
| 0 | PowerState/running | 1995 |
| 1 | PowerState/deallocated | 366 |
| 2 | PowerState/stopped | 25 |
Pivot functions [Ian]#
Pivot functions are methods of entities that provide:
data queries related to an entity
enrichment functions relevant to that entity
Pivot functions are dynamically attached to entities. We created this framework to make it easier to find which functions you can use for which entity type.
Motivation#
We had built a lot of functionality in MSTICPy for querying and enrichment
A lot of the functions had inconsistent type/parameter signatures
There was no easy discovery mechanism for these functions - you had to know
Using entities as pivot points is a “natural” investigation pattern
Access functionality from entities#
pivot = Pivot(namespace=globals())
from msticpy.datamodel import entities
display(entities.IpAddress.whois("38.75.137.9"))
display(entities.IpAddress.geoloc("38.75.137.9"))
| asn | asn_cidr | asn_country_code | asn_date | asn_description | asn_registry | nets | nir | query | raw | raw_referral | referral | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 63023 | 38.75.136.0/23 | US | 1991-04-16 | AS-GLOBALTELEHOST, US | arin | [{'cidr': '38.0.0.0/8', 'name': 'COGENT-A', 'handle': 'NET-38-0-0-0-1', 'range': '38.0.0.0 - 38.... | None | 38.75.137.9 | None | None | None |
| CountryCode | CountryName | State | City | Longitude | Latitude | Asn | TimeGenerated | Type | AdditionalData | IpAddress | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | US | United States | California | Los Angeles | -118.2441 | 34.0544 | None | None | geolocation | {} | 38.75.137.9 |
pivot.browse()
Inputs can be single values, lists or DataFrames#
%%ioc --out ip_list
SourceIP DestinationIP TotalBytesSent nir asn_registry asn asn_cidr asn_country_code asn_date asn_description query nets raw referral raw_referral
0 10.0.3.5 40.124.45.19 621 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
1 10.16.12.1 40.124.45.19 1004 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2 10.4.5.12 13.71.172.130 247 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
3 10.4.5.12 40.77.232.95 189 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
4 10.4.5.16 13.71.172.130 46 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
5 10.4.5.16 65.55.44.109 120 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
6 10.90.78.142 104.43.212.12 12 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
7 10.90.78.71 104.43.212.12 4 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
8 20.185.182.48 38.75.137.9 8328 NaN arin 8075
[('ipv4',
['10.4.5.12',
'40.77.232.95',
'10.90.78.71',
'10.90.78.142',
'65.55.44.109',
'10.0.3.5',
'13.71.172.130',
'38.75.137.9',
'40.124.45.19',
'10.4.5.16',
'20.185.182.48',
'104.43.212.12',
'10.16.12.1'])]
entities.IpAddress.whois(ip_list["ipv4"]) #, join="left")
| nir | asn_registry | asn | asn_cidr | asn_country_code | asn_date | asn_description | query | nets | raw | referral | raw_referral | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 1 | NaN | arin | 8075 | 40.76.0.0/14 | US | 2015-02-23 | MICROSOFT-CORP-MSN-AS-BLOCK, US | 40.77.232.95 | [{'cidr': '40.74.0.0/15, 40.76.0.0/14, 40.125.0.0/17, 40.124.0.0/16, 40.120.0.0/14, 40.112.0.0/1... | NaN | NaN | NaN |
| 2 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 3 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 4 | NaN | arin | 8075 | 65.52.0.0/14 | US | 2001-02-14 | MICROSOFT-CORP-MSN-AS-BLOCK, US | 65.55.44.109 | [{'cidr': '65.52.0.0/14', 'name': 'MICROSOFT-1BLK', 'handle': 'NET-65-52-0-0-1', 'range': '65.52... | NaN | NaN | NaN |
| 5 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 6 | NaN | arin | 8075 | 13.64.0.0/11 | US | 2015-03-26 | MICROSOFT-CORP-MSN-AS-BLOCK, US | 13.71.172.130 | [{'cidr': '13.104.0.0/14, 13.64.0.0/11, 13.96.0.0/13', 'name': 'MSFT', 'handle': 'NET-13-64-0-0-... | NaN | NaN | NaN |
| 7 | NaN | arin | 63023 | 38.75.136.0/23 | US | 1991-04-16 | AS-GLOBALTELEHOST, US | 38.75.137.9 | [{'cidr': '38.0.0.0/8', 'name': 'COGENT-A', 'handle': 'NET-38-0-0-0-1', 'range': '38.0.0.0 - 38.... | NaN | NaN | NaN |
| 8 | NaN | arin | 8075 | 40.124.0.0/16 | US | 2015-02-23 | MICROSOFT-CORP-MSN-AS-BLOCK, US | 40.124.45.19 | [{'cidr': '40.125.0.0/17, 40.96.0.0/12, 40.112.0.0/13, 40.120.0.0/14, 40.74.0.0/15, 40.76.0.0/14... | NaN | NaN | NaN |
| 9 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 10 | NaN | arin | 8075 | 20.184.0.0/13 | US | 2017-02-22 | MICROSOFT-CORP-MSN-AS-BLOCK, US | 20.185.182.48 | [{'cidr': '20.184.0.0/13, 20.180.0.0/14', 'name': 'MSFT', 'handle': 'NET-20-180-0-0-1', 'range':... | NaN | NaN | NaN |
| 11 | NaN | arin | 8075 | 104.40.0.0/13 | US | 2014-05-07 | MICROSOFT-CORP-MSN-AS-BLOCK, US | 104.43.212.12 | [{'cidr': '104.40.0.0/13', 'name': 'MSFT', 'handle': 'NET-104-40-0-0-1', 'range': '104.40.0.0 - ... | NaN | NaN | NaN |
| 12 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
Creating Pivot pipelines#
list(ip_list["ipv4"])[:4]
['10.4.5.12', '40.77.232.95', '10.90.78.71', '10.90.78.142']
(
entities.IpAddress.whois(list(ip_list["ipv4"])[:4], join="left")
.mp_pivot.run(entities.IpAddress.geoloc, input_col="ip_column", join="left")
.mp_pivot.run(entities.IpAddress.tilookup_ipv4, input_col="ip_column", join="left")
)
Notebooklets - “Macros” for Notebooks [Ian]#
We built notebooklets because life is too short keep writing (copy/pasting) the same code over and over again.
The Notebooklets (MSTICNB) package multiple notebook cells for common investigation routines into simple functions
Repo: https://github.com/microsoft/msticnb
Docs: https://msticnb.readthedocs.io/
$ pip install msticnb
# Import and initialize MSTIC Notebooklets - companion package
# more later
import msticnb as nb
qry_prov_az.connect(WorkspaceConfig(workspace="CyberSecuritySoc"))
nb.init(query_provider=qry_prov_az)
# qry_prov_az.connect(WorkspaceConfig(workspace="CyberSecuritySoc"))
nb.browse()
host_time = nbwidgets.QueryTime()
host_time
host_summary = nb.nblts.azsent.host.HostSummary()
host_summary_rslt = host_summary.run(value="WORKSTATION6", timespan=host_time)#, options=["-bookmarks", "-azure_api"])
host_summary_rslt.browse_alerts()
host_summary_rslt.host_entity.qry_wevt_processes(start="2021-11-17 16:00", end="2021-11-17 16:20").mp_plot.timeline(group_by="Account")
host_summary_rslt.host_entity.qry_wevt_processes(start="2021-11-17 16:09", end="2021-11-17 16:10").mp_plot.process_tree(legend_col="Account")
MSTICPy Community Sprint - Jan 2022#
MSTICPy is always open to contributions from the community, whether this be fixes to the current code base, feature additions, or just new ideas and suggestions. However, we know that contributing to an Open Source project can be a bit daunting, especially if it’s not something you have done before.
To help people with this we are running a Community Sprint during January 2022.
During this sprint we are encouraging people to engage with and contribute to MSTICPy. Contributions can take any form but in order to make this as easy as possible for people we will be offering support and guidance during the month to help people work out where and how to contribute. We will provide:
A set of contribution ideas tailored to differing skill levels and time commitments
Office Hours where you can come and ask questions and get help from the MSTICPy team
Additional contribution resources and guidance
Some awesome swag for people who contribute
Want to get involved? Keep an eye on the MSTICPy GitHub page for updates on the Community Sprint - microsoft/msticpy
Or follow us on twitter for news: @ianhellen, @MSSPete, @AshwinPatil
End of Session, End of the Workshop!!!#
Lunch break: 45 Minutes#
