AvasDream/log-analysis.md

## log-analysis.md

      
    Raw
  

              log-analysis.md
            
          
    Analyse SSH Logins

We are looking at the auth.log file in /var/log/auth.log.
All the login attempts were captured in a time frame of ~3 Hours.
Get all Ips from logfile.
sudo cat /var/log/auth.log | grep "Failed password" | awk -F 'from' '{print $2}' | cut -d" " -f2  > failed-login-ips.txt
cat failed-login-ips.txt | uniq > ips.txt
Compare total amount of IPs to uniq number:
cat failed-login-ips.txt | wc -l && cat failed-login-ips.txt | uniq | wc -l
Example Output:
5807
4689

Fail2Ban

The configuration of fail2ban is really simple.
If not already there create a local jail file:
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
Add this to the config file under the SSH section.
[sshd]

enabled	= true
port    = ssh
filter	= sshd
logpath	= /var/log/auth.log
maxretry = 4

Restart the service.
sudo service fail2ban restart
The problem with Fail2Ban is from 5807 login attempts in 3 hours, 4689 came from unique IPs.
So there was only one login attempt from this IP and our blockade of the IP will not take place.
To solve the problem, we simply change the default port of SSH.
Change SSH default port

sudo nano /etc/ssh/sshd_config
Change Entry
# Port 22

to
Port 1337

Do not use an assigned port!
Restart the Service.
sudo service sshd restart

Automate creation of IP files

Script:
#!/bin/bash

cat /var/log/auth.log | grep "Failed password" | awk -F 'from' '{print $2}' | cut -d" " -f2 | \
uniq 2>&1 > /home/vgrimmeisen/log/$(date +%d-%m-%y-%H:%M:%S)_ips.txt
cp /var/log/auth.log /home/vgrimmeisen/log/$(date +%d-%m-%y-%H:%M:%S)_auth.log
Add to crontab to execute one time per day at 3am.
crontab -e
Insert
0 3 * * * /path/to/script/monitor.sh

To test crontab you can execute it every minute:
*/1 * * * * /path/to/script/monitor.sh

Be aware that this will be executed with the User permissions of the User you are currently. Only root can read /var/log/auth.log!
Create Python script to visualize data

First script uses the ip-api to create json data files for every ip.
The API has a batch endpoint were you can query up to 100 Ip adresses in one post request.
Result:
[{
    "as": "AS4837 CHINA UNICOM China169 Backbone", 
    "city": "Hefei", 
    "country": "China", 
    "countryCode": "CN", 
    "isp": "CNC Group CHINA169 AnHui province network", 
    "lat": 31.8206, 
    "lon": 117.227, 
    "org": "", 
    "query": "58.242.83.30", 
    "region": "AH", 
    "regionName": "Anhui", 
    "status": "success", 
    "timezone": "Asia/Shanghai", 
    "zip": ""
}]
#! python
import requests
import json
import codecs

def getInfo(ip):
    resp = requests.get('http://ip-api.com/json/' + ip)
    if resp.status_code != 200:
        data_formated = "Error while getting Data for ", ip
    else:
        data = resp.json()
        # "ctry": data["country"], "isp": data["isp"],"org": data["org"]
        data_formated = {"ip": ip, "lon": data["lon"], "lat": data["lat"], "country": data["country"], "isp": data["isp"],"org": data["org"] }
    return data_formated

def getIps():
    f = open('ips.txt','r')
    ips = []
    for line in f:
        ips.append(line.rstrip("\n"))
        #print(line.rstrip("\n"))
    return ips

def generatePayload(ips):
    payload = []
    for ip in ips:
        payload.append({"query": ip})
    return payload

def batchRequest(ips):
    payload = generatePayload(ips)
    resp = requests.post('http://ip-api.com/batch', data=json.dumps(payload))
    if resp.status_code != 200:
        data_formated = "Error while getting Data for ", ip
    if resp.status_code == 422:
        data_formated = "Too much IPs in request"
    else:
        data = resp.json()
    return data


def main():
    ips = getIps()
    ip_arry = []
    # Cut array in chunks because of the api limitation
    chunks = [ips[x:x+100] for x in range(0, len(ips), 100)]
    for c,l in enumerate(chunks):

        data = batchRequest(l)
        s =  str(c)
        s += "_output.json"
        with open(s, "wb") as outfile:
            json.dump(data,codecs.getwriter('utf-8')(outfile), ensure_ascii=False)
Visualization Script
#! python
import json
import itertools
from collections import Counter
import matplotlib.pyplot as plt
import numpy as np

def readIps():
    data = {}
    for i in range(0,47):
        s =  str(i)
        s += "_output.json"
        varForI = i
        with open(s,"r", encoding="utf8") as infile:
            raw = json.load(infile)
            for c,i in enumerate(raw):
                #print(i)
                s = str(varForI)
                s += "-"
                s += str(c)
                data[s] = i
    return data

def createBarGraph(data,  xlabel, ylabel, title):
    label = []
    no = []

    for i in data:
        label.append(i[0])
        no.append(i[1])

    index = np.arange(len(label))
    plt.bar(index, no)
    plt.xlabel(xlabel, fontsize=10)
    plt.ylabel(ylabel, fontsize=10)
    plt.xticks(index, label, fontsize=10, rotation=30)
    plt.title(title)
    plt.figure(figsize=(10,6))
    plt.plot()
    return plt


def main():
    # ToDo
    # - Save to PDF
    # - Create World Map
    data = readIps()
    countrys = []
    lonlat = []
    isps = []
    # Prepare Data
    for i in data:
        countrys.append(data[i]["country"])
        lonlat.append((data[i]["lon"],data[i]["lat"]))
        isps.append(data[i]["isp"])
    # Count Data
    ispsCount = Counter(isps).most_common()
    countrysCount = Counter(countrys).most_common()
    # Top Ten
    countryTop10 = countrysCount[:10]
    ispsTop10 = ispsCount[:10]
    # Create Bar Graph
    countryPlt = createBarGraph(countryTop10, 'Countrys', 'No of Countrys', 'Source of IP adresses')
    ispPlt = createBarGraph(ispsTop10, 'ISP', 'No of ISP', 'ISP of IP adresses')
    
    # Save to PDF
    
    # Create World Map
    
if __name__ == "__main__":
    main()
No results found