Skip to content

Instantly share code, notes, and snippets.

@cmd-not-found

cmd-not-found/synapse-bootstrap.md

Last active August 26, 2020 02:21
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save cmd-not-found/a684b7b3e6ab99de8b1491a90e101ed8 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save cmd-not-found/a684b7b3e6ab99de8b1491a90e101ed8 to your computer and use it in GitHub Desktop.
Download ZIP
Start Using Vertex Synapse Project Locally
Raw
synapse-bootstrap.md

GETTING STARTED WITH VERTEX SYNAPSE

These are some rough notes collected via MacDown while exploring the latest v2.7.0 of the Vertex Synapse Project. Just starting from scratch in a local dev environment on my Mac.

SET UP DEV ENVIRONMENT

This makes some assumptions that you have pyenv installed to manage local Python versions. Using python 3.7.x since the project is based on 3.7 - uncertain of support for 3.8+.

jberry@Metaverse projects % mkdir synapse-dev 
jberry@Metaverse projects % cd synapse-dev 
jberry@Metaverse synapse-dev % pyenv local 3.7.8
jberry@Metaverse synapse-dev % python -m venv ./venv
jberry@Metaverse synapse-dev % source ./venv/bin/activate
(venv) jberry@Metaverse synapse-dev % pip install synapse
<snip>

STARTING SYNAPSE CORTEX WITH DOCKER

Published docker containers are probably the easiest way to get started.

jberry@Metaverse synapse % docker pull vertexproject/synapse:v2.7.0
v2.7.0: Pulling from vertexproject/synapse
6ec8c9369e08: Pull complete 
<snip>
6f0198934bcd: Pull complete 
Digest: sha256:0cf8838ce66f6d7805fa447ca4422b801a81b6388a89bbca60e443cbaefcdfd5
Status: Downloaded newer image for vertexproject/synapse:v2.7.0
docker.io/vertexproject/synapse:v2.7.0
jberry@Metaverse synapse % docker pull vertexproject/synapse-cortex:v2.7.0 
v2.7.0: Pulling from vertexproject/synapse-cortex
6ec8c9369e08: Already exists 
<snip>
90b84f28234c: Pull complete 
Digest: sha256:c9e2458bf8f156dd393f93f66f337719c60a9c585456c1ccb0fbd7ef878010b9
Status: Downloaded newer image for vertexproject/synapse-cortex:v2.7.0
docker.io/vertexproject/synapse-cortex:v2.7.0

UPDATE docker-compose.yml AND RUN CONTAINER

REF: https://synapse.docs.vertex.link/en/latest/synapse/quickstart.html#using-docker

NOTE: I'm mapping my local directory /Users/jberry/projects/synapse-dev/cortex-docker to the /vertex/storage on the container itself, making the data persistent beyond the docker container state.

(venv) jberry@Metaverse synapse % cat docker-compose.yml 
version: '3'

services:

    core00:

        image: vertexproject/synapse-cortex:v2.7.0

        volumes:
            # Map in a persistent storage directory
            - /Users/jberry/projects/synapse-dev/cortex-docker:/vertex/storage

        environment:
            # Set a default password for the root user
            - SYN_CORTEX_AUTH_PASSWD=secret

        ports:
            # Default https port
            - "4443:4443"
            # Default telepath port
            - "27492:27492"
(venv) jberry@Metaverse synapse %

NOTE: The password secret will be the root user's default password for the docker container as defined by the SYN_CORTEX_AUTH_PASSWD.

Then bring the container up. The -d runs it in detached mode so it'll stay running in the background.

(venv) jberry@Metaverse synapse-dev % mkdir cortex-docker
(venv) jberry@Metaverse synapse-dev % docker-compose up -d 
Creating network "synapse_default" with the default driver
Creating synapse_core00_1 ... done
(venv) jberry@Metaverse synapse % docker container ls 
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                   PORTS                                              NAMES
5873ffce31b8        vertexproject/synapse-cortex:v2.7.0   "/bin/sh -c /vertex/…"   6 minutes ago       Up 6 minutes (healthy)   0.0.0.0:4443->4443/tcp, 0.0.0.0:27492->27492/tcp   synapse_core00_1

To stop the container and restart:

(venv) jberry@Metaverse synapse % docker-compose down
Stopping synapse_core00_1 ... done
Removing synapse_core00_1 ... done
Removing network synapse_default
(venv) jberry@Metaverse synapse % docker-compose up -d               
Creating network "synapse_default" with the default driver
Creating synapse_core00_1 ... done
(venv) jberry@Metaverse synapse %

CONNECT TO RUNNING CORTEX VIA TELEPATH

To connect to the cortex via telepath (built-in mecahnism for Synapse), which requires the venv enabled and/or the synapse package installed locally, use the following command to to connect over port 27492.

(venv) jberry@Metaverse synapse-dev % docker-compose up -d
Creating network "synapse-dev_default" with the default driver
Creating synapse-dev_core00_1 ... done
(venv) jberry@Metaverse synapse-dev % python -m synapse.tools.cmdr tcp://root:secret@localhost:27492/  
cli> storm syn:form=inet:ipv4
Executing query at 2020/08/26 01:04:59.224
syn:form=inet:ipv4
        :doc = An IPv4 address.
        :runt = False
        :type = inet:ipv4
complete. 1 nodes in 18 ms (55/sec).
cli> quit
o/
(venv) jberry@Metaverse synapse-dev %

CONNECT TO RUNNING CORTEX VIA REST API

REF: https://synapse.docs.vertex.link/en/latest/synapse/httpapi.html#authentication

Using Postman, executing a POST request to https://localhost:4443/api/v1/login with the following body:

{"user": "root", "passwd": "secret"}

Returned 200 status code with a sess cookie and the following response body:

{
    "status": "ok",
    "result": {
        "type": "user",
        "iden": "727f0bd01350ce064c755b6d5782eb05",
        "name": "root",
        "rules": [],
        "roles": [],
        "admin": true,
        "email": null,
        "locked": false,
        "archived": false,
        "authgates": {
            "76337ca7d579d4a94e09ed7752d737bf": {
                "admin": true
            },
            "06733d52ae90a00376a16bddcca093bf": {
                "admin": true
            }
        }
    }
}

STARTING SYNAPSE CORTEX LOCALLY

Need to set up the local dev environment (i.e. a venv) and install synapse package first.

CREATE A LOCAL CORTEX USING synapse.servers.cortex

In a separate command prompt, start the cortex server.

(venv) jberry@Metaverse synapse-dev % python -m synapse.servers.cortex ./cortex
2020-08-25 21:39:09,378 [INFO] log level set to INFO [common.py:setlogging:MainThread:MainProcess]
...cortex API (telepath): tcp://0.0.0.0:27492
...cortex API (https): 4443

Then, from the working directory, you can simply connect using the cell://cortex syntax without having to connect "remotely".

(venv) jberry@Metaverse synapse-dev % python -m synapse.tools.cmdr cell://cortex              
cli>
cli>
cli> storm syn:form=inet:ipv4
Executing query at 2020/08/26 01:41:35.579
syn:form=inet:ipv4
        :doc = An IPv4 address.
        :runt = False
        :type = inet:ipv4
complete. 1 nodes in 13 ms (76/sec).
cli>

You can also connect via the remote methods listed above if you set and/or enable the root user's password.

REF: https://synapse.docs.vertex.link/en/latest/synapse/quickstart.html#remote-access

(venv) jberry@Metaverse synapse-dev % python -m synapse.tools.cellauth cell://cortex modify root --passwd secret     
2020-08-25 21:45:38,418 [INFO] log level set to DEBUG [common.py:setlogging:SynLoop:MainProcess]
setting passwd for: root
root (1e482e11e5f70ebc57f55beb7ef577a2)
type: user
admin: True
locked: False
rules:
  auth gate: 00cc7bbb41fda7b47cef88d1d97235f2
  auth gate: a1ce886bf68a955489a05b6e1c95eaa0

roles:
(venv) jberry@Metaverse synapse-dev % python -m synapse.tools.cmdr tcp://root:secret@localhost:27492/
cli>
cli>

CUSTOM SYNAPSE MODULE

REF: Documentation provided here: https://synapse.docs.vertex.link/en/latest/synapse/devguides/cortex_quickstart.html#basics

SETUP ENV AND CREATE LOCAL CORTEX

Started by using python 3.7.8 in a new dir and created a local cortex in my working directory for testing.

jberry@Metaverse projects % mkdir synapse2
jberry@Metaverse projects % cd synapse2 
jberry@Metaverse synapse2 % pyenv local 3.7.8
jberry@Metaverse synapse2 % python -m venv ./venv
jberry@Metaverse synapse2 % source ./venv/bin/activate
(venv) jberry@Metaverse synapse2 % pip install synapse
Collecting synapse
<snip>
(venv) jberry@Metaverse synapse2 % python -m synapse.servers.cortex ./cortex
2020-08-23 21:17:09,414 [INFO] log level set to INFO [common.py:setlogging:MainThread:MainProcess]
...cortex API (telepath): tcp://0.0.0.0:27492
2020-08-23 21:17:13,754 [WARNING] NO CERTIFICATE FOUND! generating self-signed certificate. [cell.py:addHttpsPort:MainThread:MainProcess]
...cortex API (https): 4443

^CCaught SIGINT, shutting down.

INSTALL example PYTHON MODULE

After creating the local cortex, I installed an example python module.

REF: I followed these instructions to create the most bare-bones python package. https://python-packaging.readthedocs.io/en/latest/minimal.html

jberry@Metaverse synapse2 % tree ./example 
./example
├── example
│   ├── __init__.py
└── setup.py
__init__.py

This is simply going to print hello cortex! on startup.

import synapse.lib.module as s_module

class ExampleModule(s_module.CoreModule):

    async def initCoreModule(self):
        # by this time we have a reference to the Cortex as self.core
        print(f'hello cortex!')
setup.py
from setuptools import setup

setup(name='example',
      version='0.1',
      description='The cortex example',
      url='https://synapse.docs.vertex.link/en/latest/synapse/devguides/cortex_quickstart.html',
      packages=['example'],
      zip_safe=False)

Then, cd to the first directory (the one with setup.py in it) and execute the following command:

NOTE: If you're using a venv, ensure it's activated.

(venv) jberry@Metaverse synapse2 % cd ./example
(venv) jberry@Metaverse example % pip install .
Processing /Users/jberry/projects/synapse2/example
Using legacy 'setup.py install' for example, since package 'wheel' is not installed.
Installing collected packages: example
    Running setup.py install for example ... done
Successfully installed example-0.1

UPDATE cell.yaml CONFIG TO LOAD MODULE

Since we're using the ./cortex folder in our working directory, you'll navigate to that folder and create a new file called cell.yaml.

(venv) jberry@Metaverse cortex % vi cell.yaml
(venv) jberry@Metaverse cortex % ls -lsa
total 64
 0 drwx------  13 jberry  staff   416 Aug 25 22:16 .
 0 drwxr-xr-x  11 jberry  staff   352 Aug 25 21:03 ..
24 -rw-r--r--@  1 jberry  staff  8196 Aug 23 23:23 .DS_Store
 0 drwx------  12 jberry  staff   384 Aug 25 21:39 axon
 8 -rw-r--r--   1 jberry  staff    32 Aug 23 21:16 cell.guid
 8 -rw-r--r--   1 jberry  staff    65 Aug 23 23:04 cell.yaml
 0 drwx------   5 jberry  staff   160 Aug 23 21:16 certs
 8 -rw-r--r--   1 jberry  staff    32 Aug 23 21:16 cookie.secret
 0 drwxr-xr-x   4 jberry  staff   128 Aug 23 23:22 layers
 0 drwx------   9 jberry  staff   288 Aug 23 23:22 slabs
 0 srwxr-xr-x   1 jberry  staff     0 Aug 25 21:39 sock
 8 -rw-r--r--   1 jberry  staff  1797 Aug 23 21:17 sslcert.pem
 8 -rw-r--r--   1 jberry  staff  3268 Aug 23 21:17 sslkey.pem

cell.yaml

modules:
    - example.ExampleModule

RESTART CORTEX TO SEE PRINTED STATEMENT

(venv) jberry@Metaverse synapse2 % python -m synapse.servers.cortex ./cortex
2020-08-23 21:20:05,524 [INFO] log level set to INFO [common.py:setlogging:MainThread:MainProcess]
hello cortex!
...cortex API (telepath): tcp://0.0.0.0:27492
...cortex API (https): 4443

The hello cortex! is what was defined in the custom ExampleModule class. w00t!

CUSTOM SYNAPSE DATAMODEL

Taking custom development a little bit further by creating a custom data model using the example provided here:

REF: https://synapse.docs.vertex.link/en/latest/synapse/devguides/cortex_quickstart.html#data-model-extensions

Similar setup as above except different module name c_model.

jberry@Metaverse synapse2 % tree ./c_model 
./c_model
├── c_model
│   ├── __init__.py
└── setup.py
__init__.py
import synapse.lib.module as s_module

class ExampleModule(s_module.CoreModule):

    def getModelDefs(self):

        # we return a tuple of (name, modeldef) tuples...

        return (

            ('foomodel', {

                'types': (

                    # declare a type for our form primary property
                    ('x:foo:event', ('str', {'regex': '[a-z]{2}-[0-9]{5}'}), {
                        'doc': 'A custom event ID from some other system.'}),

                    ('x:foo:bar', ('int', {'min': 100, 'max': 10000}), {
                        'doc': 'A custom integer property with a fixed range.'}),
                ),

                'forms': (

                    # declare a new node type

                    ('x:foo:event', {}, (

                        # declare secondary properties
                        ('time', ('time', {}), {
                            'doc': 'The time of the custom event.'}),

                        ('ipv4', ('inet:ipv4', {}), {
                            'doc': 'The ipv4 associated with the custom event.'}),

                        ('bar', ('x:foo:bar', {}), {
                            'doc': 'The custom bar property associated with the custom event.'}),
                    )),
                ),

            }),
        )
setup.py
from setuptools import setup

setup(name='c_model',
      version='0.1',
      description='The cortex example',
      url='https://synapse.docs.vertex.link/en/latest/synapse/devguides/cortex_quickstart.html',
      packages=['c_model'],
      zip_safe=False)

UPDATE CONFIG

Make sure you update the cell.yaml config with the latest configuration to include the new package c_model.

modules:
    - c_model.ExampleModule
    - example.ExampleModule

Same steps to install the module via pip from within the venv. Then, start the cortex server (maybe in a separate terminal or in the background).

(venv) jberry@Metaverse synapse2 % cd ./c_model
(venv) jberry@Metaverse c_model % pip install .
Processing /Users/jberry/projects/synapse-dev/c_model
Using legacy 'setup.py install' for c-model, since package 'wheel' is not installed.
Installing collected packages: c-model
  Attempting uninstall: c-model
    Found existing installation: c-model 0.1
    Uninstalling c-model-0.1:
      Successfully uninstalled c-model-0.1
    Running setup.py install for c-model ... done
Successfully installed c-model-0.1
(venv) jberry@Metaverse c_model % cd ../
(venv) jberry@Metaverse c_model % python -m synapse.servers.cortex ./cortex
2020-08-23 22:41:53,983 [INFO] log level set to INFO [common.py:setlogging:MainThread:MainProcess]
hello cortex!
...cortex API (telepath): tcp://0.0.0.0:27492
...cortex API (https): 4443

Now, we can validate the new data model took effect by simply querying for each one in the loaded datamodel:

  • cli> storm syn:form=x:foo:event
  • cli> storm syn:type=x:foo:bar
  • cli> storm syn:type=x:foo:event
(venv) jberry@Metaverse c_model % python -m synapse.tools.cmdr cell://cortex 
cli> storm syn:form=x:foo:event
Executing query at 2020/08/24 02:42:33.180
syn:form=x:foo:event
        :doc = A custom event ID from some other system.
        :runt = False
        :type = x:foo:event
complete. 1 nodes in 20 ms (50/sec).
cli> storm syn:type=x:foo:bar
Executing query at 2020/08/24 03:00:00.789
syn:type=x:foo:bar
        :ctor = synapse.lib.types.Int
        :doc = A custom integer property with a fixed range.
        :opts = {'size': 8, 'signed': True, 'fmt': '%d', 'min': 100, 'max': 10000, 'ismin': False, 'ismax': False}
        :subof = int
complete. 1 nodes in 20 ms (50/sec).
cli> storm syn:type=x:foo:event
Executing query at 2020/08/24 03:00:11.290
syn:type=x:foo:event
        :ctor = synapse.lib.types.Str
        :doc = A custom event ID from some other system.
        :opts = {'enums': None, 'regex': '[a-z]{2}-[0-9]{5}', 'lower': False, 'strip': False, 'replace': (), 'onespace': False, 'globsuffix': False}
        :subof = str
complete. 1 nodes in 17 ms (58/sec).

Now, we can try interacting with the new nodes and create a node of the new model type, and then execute a similar query as is provided in the docs:

REF: https://synapse.docs.vertex.link/en/latest/synapse/devguides/cortex_quickstart.html#data-model-extensions

(venv) jberry@Metaverse synapse-dev % python -m synapse.tools.cmdr cell://cortex                     
cli> storm [ x:foo:event=ab-98765 :ipv4=10.10.10.10 :bar=200 :time=20200822 ]
Executing query at 2020/08/24 02:43:00.971
........
x:foo:event=ab-98765
        .created = 2020/08/24 02:43:00.993
        :bar = 200
        :ipv4 = 10.10.10.10
        :time = 2020/08/22 00:00:00.000
complete. 1 nodes in 27 ms (37/sec).
cli> storm inet:ipv4=10.10.10.10
Executing query at 2020/08/24 02:43:20.891
inet:ipv4=10.10.10.10
        .created = 2020/08/24 02:43:00.994
        :type = private
complete. 1 nodes in 13 ms (76/sec).
cli> storm x:foo:event = ab-98765
Executing query at 2020/08/24 02:43:58.450
x:foo:event=ab-98765
        .created = 2020/08/24 02:43:00.993
        :bar = 200
        :ipv4 = 10.10.10.10
        :time = 2020/08/22 00:00:00.000
complete. 1 nodes in 17 ms (58/sec).
cli> storm --hide-props x:foo:event:time*range=(20200821, 20200823) -> inet:ipv4
Executing query at 2020/08/24 02:45:00.779
inet:ipv4=10.10.10.10
complete. 1 nodes in 29 ms (34/sec).
cli>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment