自动化运维开发-ansible接口

目录​

​​探测模块和工具 2​​​

​​存活扫描nmap|telnetlib 2​​​

​​主机登录探测pexpect|paramiko 2​​​

​​ansible运维 4​​​

​​ansible开发： 5​​​

​​核心类 5​​​

​​ad-hoc模式调用 6​​​

​​playbook模式调用 7​​​

​​callback改写 7​​​

​​自动化任务接口设计 9​​​

​​数据库事件记录和状态记录 9​​​

自动化资产扫描发现（服务端扫描发现）：​

1、抽象与约定：​

主机类型，Centos4-6、Ubuntu12-14；​

判断是linux系统，22、202、2022；​

安全规则，开放允许探测协议和登录的限制；​

网络设备都开通了snmp服务，且community都已经统一；​

虚拟机不再运行容器等虚拟资产；​

2、整体探测流程：​

存活探测-->获取存活的ip列表；​

主机探测-->获取系统信息（SN、系统版本(cat /etc/issue+cat /etc/redhat-release|uname|lsb_release)、MAC(cat /sys/class/net/eth*/address|ifconfig eth0|ip a|esxcfg-vmknic -l)、主机名(hostname|uname -a|cat /etc/sysconfig/network)、服务器机型(dmidecode -s system-manufacturer|dmidecode -s system-product-name|dmidecode -s system-serial-number)）；​

主机关系探测-->识别宿主机和虚拟机关系；​

网络设备探测-->网络设备信息（SN、设备名等）；​

cat /sys/class/net/[^vtlsb]*/address # 获取mac，排除v|t|l|s|b开头的​

esxcfg-vmknic -l | awk '{print $8}' | grep ':' # esxi主机​

cat /sys/class/net/[^vtlsb]*/address || esxcfg-vmknic -l | awk '{print $8}' | grep ':' # 前一条未执行成功再执行后一条​

yum -y install dmidecode​

探测协议：​

特性​

用途​

icmp​

无连接​

网络探测、网络质量​

tcp​

有连接​

应用服务​

资产扫描：​

服务器资产信息：硬件服务器、kvm服务器、esx虚拟机；​

未知设备ip列表：网络设备、其它设备；​

探测模块和工具​

存活扫描nmap|telnetlib​

nmap，是一款用于网络发现和安全审计的网络安全工具，pip install python-nmap==0.6.1；​

nmap -n -sP 192.168.8.70 # arp，在局域网内仅发送一个包，询问谁是8.70，在整个网段扫描时效率高，而ping发送2个包且是icmp；192.168.8.70上运行tcpdump -np -i eth0 src host 192.168.8.119；echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_all​

nmap -n -PE 192.168.8.70 # tcp，判断该主机哪些端口存活​

nmap -n -sP -PE 192.168.1.0/24 # arp和tcp都探测，从多个角度判断服务端的存活状态，且准确（即使服务端忽略或关闭icmp）​

import nmap​

nm = nmap.PortScanner()​

nm.scan(hosts='192.168.1.0/24', arguments="-n -sP -PE")​

nm.all_hosts() # ip列表​

import telnetlib​

tm = telnetlib.Telnet(host='192.168.1.101', port='22', timeout=4)​

txt = tm.read_until('\n', timeout=5) # 检测到换行​

re.search('ssh', txt)​

主机登录探测pexpect|paramiko​

用一系列的验证方式循环进行ssh登录，得到正确的登录方式；​

ssh -l root 192.168.1.101 -p 22 # 账号密码登录​

ssh -i /tmp/id_rsa -l test 192.168.1.101 -p 22 # 密钥登录，私钥​

pexpect，用来通过启动子程序，使用正则对程序输出作出特定响应，以此实现与其自动交互的py模块；​

缺陷，依赖终端命令的方式，不同的ssh登录环境兼容较差；​

核心类、函数：​

run()，直接进程运行，直接返回结果和状态；​

spawn()，启动子进程运行，有丰富的方法实现对子程序的控制，读取缓冲区，正则匹配成功（1发送指令（send|sendline|sendcontrol(char)）读取缓冲区进入循环，2让出子进程会话终端接管（intract）进入终端交互模式），正则匹配不成功（timeout等待超时|pexpect.EOF子程序退出），打印输出匹配的缓冲区结果（before|after）​

import pexpect​

pexpect.run(command='ls /tmp', withexiststatus=1) # 命令执行后返回的结果和状态，二元组，1成功0失败​

ssh = pexpect.spanwn('ssh root@192.168.1.101 -p22') # 类的实例化，chk = pexpect.spawn(' ls -l /tmp/')同chk = pexpect.spawn('ls', ['-l', '/tmp/'])​

i = ssh.expect(['password:', 'continue connecting (yes/no)?'], timeout=5) # 缓冲区内容匹配，匹配成功返回0，没有匹配到则等待子程序超时，默认30s；支持正则ssh.expect('[p,P]assword:')，$在expect中就是本身的意义，不是正则中的结尾；匹配多个结果，ssh.expect([pexpect.TIMEOUT,pexpect.EOF,'password:'])，返回匹配列表中的索引号，如匹配到了'password:'则返回2​

if i == 0:​

向子程序发送指令​

elif i == 1:​

ssh.sendline('yes\n')​

ssh.expect('password: ')​

ssh.sendline('pwd')​

index = ssh.expect(['#', pexpect.EOF, pexpect.TIMEOUT])​

if index == 0:​

print('logging in as root')​

终端会话，进入终端​

elif index == 1:​

print('logging process exit')​

elif index == 2:​

print('logging timeout exit')​

paramiko，基于py实现的ssh远程安全连接，用于ssh远程执行命令、文件传输等功能的ssh客户端模块；​

import paramiko​

ssh = paramiko.SSHClient()​

ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())​

ssh.connect('192.168.1.101', '22', 'test', '123456') # 账号密码​

key = paramiko.RSAKey.from_private_key_file('/tmp/id_rsa')​

ssh.connect('192.168.1.101', '22', 'test', pkey=key) # 密钥​

stdin, stdout, stdout = ssh.exec_command('ls /tmp/')​

stdout.read()​

资产扫描-snmp网络设备​

资产扫描-docker容器扫描​

docker ps | awk -F '->' '{print $1}' | grep -v 'CONTAINER' | awk 'BEGIN{FS~/s+/;}{print $NF" "$1" "$2;}' | sed s/0.0.0.0://​

资产扫描-KVM虚拟机扫描​

是否存在进程qume-kvm|docker-containerd|vmx，vmx为vmware宿主机；​

cat /sys/class/net/vnet0/address # 虚机和宿主机的mac相同，判断哪些虚机在一物理机上​

资产扫描-SDK调用扫描ESXI​

dmidecode -s system-serial-number # 虚机的uuid，再通过sdk或api获取所拥有的虚机的uuid，https://www.vmware.com/support/pubs/sdk_pubs.html，pyvmomi==6.5.0.2017.5.post1​

ansible运维​

自动化任务（v2.4.1.0-0.4.rc2.tar.gz）：​

配置文件顺序：​

export ANSIBLE_CONFIG=/etc/ansible/ansible.cfg​

./ansible.cfg # 执行命令的当前路径​

~/ansible.cfg​

/etc/ansible/ansible.cfg​

ansible.cfg​

[defaults]​

inventory = /etc/ansible/hosts​

library = /usr/share/ansible​

forks = 5​

sudo_user = root​

remote_port = 22​

host_key_checking = False # 设置是否检查ssh主机的密钥​

timeout = 20​

log_path = /var/log/ansible.log # 默认不记录​

private_key_file = /path/to/file.pem # 用ssh私钥登录时使用的密钥路径​

/etc/ansible/hosts​

[test] # 组名​

192.168.1.11:22 ansible_ssh_user=root ansible_ssh_pass='123456' # yum -y install sshpass​

192.168.1.12:22 ansible_ssh_user=root ansible_ssh_private_key_file=/home/ssh_keys/id_rsa​

test1 ansible_ssh_host=192.168.1.13 ansible_ssh_port=22 ansible_ssh_user=root ansible_ssh_private_key_file=/home/ssh_keys/id_rsa # 别名+ssh用户+ssh私钥​

ansible <host-pattern> [options]​

ansible-playbook playbook.yml [options]​

ansible all --list-hosts​

ansible 192.168.1.* -a 'ls /tmp'​

ad-hoc模式，短简命令（临时命令），多台主机上（查看某个进程是否启动|拷贝指定日志文件到本地）；​

playbook模式；​

ansible开发：​

import ansible​

print(ansible.__version__) # 2.4.1.0​

核心类​

from ansible.parsing.dataloader import DataLoader​

from ansible.vars.manager import VariableManager​

from ansible.inventory.manager import InventoryManager​

from ansible.playbook.play import Play​

from ansible.executor.task_queue_manager import TaskQueueManager​

from ansible.plugins.callback import CallbackBase​

核心类​

用途​

所在的模块路径​

DataLoader​

读取yaml|json格式的文件​

ansible.parsing.dataloader​

Play​

存储执行hosts的角色信息​

ansible.playbook.play​

TaskQueueManager​

ansible底层用到的任务队列​

ansible.executor.task_queue_manager​

PlaybookExecutor​

核心类执行playbook剧本​

ansible.executor.playbook_executor​

CallbackBase​

状态回调，各种成功|失败的状态​

ansible.plugins.callback​

InventoryManager​

导入inventory文件​

ansible.inventory.manager​

VariableManager​

存储各类变量信息​

ansible.vars.manager​

Host,Group​

操作单个主机或主机组信息​

ansible.inventory.host​

InventoryManager --> VariableManager --> ad-hoc模式调用|playbook模式调用；​

loader = DataLoader()​

inventory = InventoryManager(loader=loader, sources=['/etc/ansible/hosts'])​

inventory.get_group_dict() # 查看主机组资源，返回字典，key为组名，value为主机列表，{'all': ['192.168.1.112', '192.168.1.113], 'test_group1': ['192.168.1.112'], 'test_group2': ['192.168.1.113']}​

inventory.get_hosts()​

inventory.add_host(host='192.168.1.122', port=22, group='test_group2') # 添加主机到指定主机组​

host=inventory.get_host(hostname='192.168.1.112') # 获取指定的主机对象​

variable = VariableManager(loader=loader, inventory=inventory)​

variable.get_vars() # 查看变量​

varialbe.get_vars(host=host)​

varialbe.set_host_variable(host=host, varname='ansible_ssh_pass', value='123456') # 设置主机变量​

varialbe.get_vars(host=host)​

variable.extra_vars={'myweb': 'test', 'myname': 'jowin'} # 设置扩展变量​

varialbe.get_vars(host=host)​

variable.get_vars()​

ad-hoc模式调用​

执行对象（命令）和模块，Play；​

资源资产配置清单，InventoryManager和VarialbeManager；​

执行选项，Options；​

loader = DataLoader()​

inventory = InventoryManager(loader=loader, source=['imoocc_hosts'])​

variable_manager = VariableManager(loader=loader, inventory=inventory)​

from collections import namedtuple​

Options = namedtuple('Options', ['connection', 'remote_user', 'ask_sudo_pass', 'verbosity', 'ack_pass', 'module_path', 'forks', 'become', 'become_method', 'become_user', 'check', 'listhosts', 'listtasks', 'listtags', 'syntax', 'sudo_user', 'sudo', 'diff'])​

options = Options(cnotallow='smart', remote_user=None, ack_pass=None, sudo_user=None, forks=5, sudo=None, ask_sudo_pass=False, verbosity=5, module_path=None, become=None, become_method=None, become_user=None, check=False, diff=False, listhosts=False, listtasks=None, listtags=None, syntax=None) # smart表示连接远程主机，local连接本地主机​

play_source = dict(name='ansible play ad-hoc test', hosts='192.168.1.110', gateher_facts='no', tasks=[dict(actinotallow=dict(module='shell', args='touch /tmp/ad_hoc_test'))])​

play = Play().load(play_source, variable_manager=varialbe_manager, loader=loader)​

passwords = dict() # hosts文件中已定义了用户名和密码​

tqm = TaskQueueManager(inventory=inventory, variable_manager=varialbe_manager, loader=loader, optinotallow=options, passwords=passwords)​

result = tqm.run(play)​

playbook模式调用​

palybook = PlayBookExecutor(playbooks=['test.yml'], inventory=inventory, variable_manager=varialbe_manager, loader=loader, optinotallow=options, passwords=passwords)​

playbook.run()​

callback改写​

原因是为了自定义输出；​

通过子类继承CallbackBase父类；​

通过子类改写父类的部分方法，如v2_runner_on_unreacheable|v2_runner_on_ok|v2_runner_on_failed；​

class ModelResultsCollector(CallbackBase):​

def __init__(self, *args, **kwargs):​

super(ModelResultsCollector, self).__init__(*args, **kwargs)​

self.host_ok = {}​

self.host_failed = {}​

self.host_unreachable = {}​

def v2_runner_on_ok(self, result, *args, **kwargs):​

self.host_ok[result._host.get_name()] = result​

def v2_runner_on_failed(self, result, *args, **kwargs):​

self.host_failed[result._host.get_name()] = result​

def v2_runner_on_unreachable(self, result):​

self.host_unreachable[result._host.get_name()] = result​

callback = ModelResultsCollector()​

tqm = TaskQueueManager(inventory=inventory, variable_manager=variable_manager, loader=loader, optinotallow=options, passwords=passwords, stdout_callback=callback)​

result = tqm.run(play)​

print(callback.host_ok.items())​

result_raw = {'success': {}, 'failed': {}, 'unreachable': {}}​

for host, result in callback.host_ok.items():​

result_raw['success'][host] = result._result​

for host, result in callback.host_failed.items():​

result_raw['failed'][host] = result._result​

for host, result in callback.host_unreachable.items():​

result_raw['unreachable'][host] = result._result​

print(result_raw)​

class PlayBookResultsCollector(CallbackBase):​

CALLBACK_VERSION = 2.0​

def __init__(self, *args, **kwargs):​

super(PlayBookResultsCollector, self).__init__(*args, **kwargs)​

self.task_ok = {}​

self.task_failed = {}​

self.task_unreachable = {}​

self.task_status = {}​

self.task_skipped = {}​

def v2_runner_on_ok(self, result, *args, **kwargs):​

self.task_ok[result._host.get_name()] = result​

def v2_runner_on_failed(self, result, *args, **kwargs):​

self.task_failed[result._host.get_name()] = result​

def v2_runner_on_reachable(self, result, *args, **kwargs):​

self.task_unreachable[result._host.get_name()] = result​

def v2_runner_on_status(self, stats):​

hosts = sorted(stats.processed.keys())​

for h in hosts:​

t = stats.summarize(h)​

self.task_status[h] = {'ok': t['ok'], 'changed': t['changed'], 'unreachable': t['unreachable'], 'skipped': t['skipped'], 'failed': t['failures']}​

def v2_runner_on_skipped(self, result):​

self.task_skipped[result._host.get_name()] = result ​

playbook = PlaybookExecutor(playbook=['test.yml'], inventory=inventory, variable_manager=variable_manager, loader=loader, optinotallow=options, passwords=passwords)​

playbook._tqm.stdout_callback = callback​

playbook.run()​

results_raw = {'skipped': {}, 'failed': {}, 'ok': {}, 'unreachable': {}}​

for host, result in callback.task_ok.items():​

results_raw['ok'][host] = result​

print(results_raw)​

自动化任务接口设计​

urls.py --> views --> util层 --> module​

view层，django逻辑视力实现任务逻辑处理；​

util层，ansible实现ad-hoc、playbook功能封装；​

taskdo/utils/ansible_api.py​

taskdo/views.py​

数据库事件记录和状态记录​

事件日志意义，提交任务时，能实时追踪任务的执行进展；​

{'taskid': self.task_id, 'time': time, 'id': id, 'desc': record_info}；​

mongo实现日志记录；​

redis任务锁功能（同一时刻只能1个任务执行）和状态记录；​

taskdo/utils/base/MgCon.py​

import pymongo​

mgc = pymongo.MongoClient('192.168.1.108', 27017)​

db = mgc['imoocc']​

db.newdata.insert_one({'aa': 11, 'bb': 22})​

db.newdata.update_one({'aa': 11}, {"$set": {'aa': 33}})​

db.newdata.find_one()​

taskdo/utils/base/RedisCon.py​

import redis​

connpool = redis.ConnectionPool(host='192.168.1.108', port=6379)​

rc = redis.Redis(connection_pool=connpool)​

rc.set('aa', 11)​

rc.get('aa')​