批量造数据程序
上一篇 /
下一篇 2017-03-26 12:33:10
/ 个人分类:python
摘要:批量造数据程序,适用于压力测试,测试数据准备,SQL查询语句优化(需要大量数据)
可一定之进程数,每个进程中开启线程数,已经进程处理的数据量等
Mr. Neo Chen (netkiller), 陈景峰(BG7NYT)
<openunix@163.
批量造数据程序,适用于压力测试,测试数据准备,SQL查询语句优化(需要大量数据)
可一定之进程数,每个进程中开启线程数,已经进程处理的数据量等
Mr. Neo Chen (netkiller), 陈景峰(BG7NYT)
下面是我多年积累下来的经验总结,整理成文档供大家参考:
主程序
-
-
-
-
-
-
-
-
- logfile = '/tmp/loopdata.log'
-
- import MySQLdb
- import Queue
- import threading
- import time
- import logging
- import os,sys
- import random, string
-
- import table
-
- class ThreadDB(threading.Thread):
- def __init__(self, queue):
- threading.Thread.__init__(self)
- self.queue = queue
- logging.basicConfig(level=logging.NOTSET,
- format='%(asctime)s %(levelname)-8s %(message)s',
- datefmt='%Y-%m-%d %H:%M:%S',
- filename=logfile,
- filemode='a')
- self.logging = logging.getLogger()
- self.logging.debug(self.name + ' Start')
-
-
- def run(self):
- db=MySQLdb.connect(host='localhost', user='neo', passwd='chen',db="neo", charset="utf8")
-
- cursor=db.cursor()
- running = True
- sql = None
- while running:
-
- try:
- sql = self.queue.get()
- except Exception as e:
- print (e)
- running=False
- break
- try:
-
- n = 0
- n = cursor.execute(sql)
- log = self.name +"\t"+ '' +''+' '+sql
- self.logging.debug(log.replace('\n','').replace('\t',''))
-
- except NameError as e:
- print (e)
- break
- except Exception as e:
- print (e)
- break
- except:
- break
-
- self.queue.task_done()
- db.commit()
- cursor.close()
- db.close()
- self.logging.debug(self.name + ' End')
-
- class Main():
- def __init__(self):
- self.queue = Queue.Queue()
- def threading(self, maxconn = 100):
- for i in range(maxconn):
- t = ThreadDB(self.queue)
- t.setDaemon(True)
- t.start()
- pass
- def run(self, count = 0):
-
- n = 0
- while n < count:
-
- self.queue.put(table.test1())
-
- n = n + 1
-
-
- self.queue.join()
-
- class Loop():
- def __init__(self, process = 1, thread = 1, count = 1):
- n = 0
- while n < process :
- try:
- pid = os.fork()
- if pid > 0:
-
- sys.exit(0)
- else:
- main = Main()
- start = time.time()
- main.threading(thread)
- main.run(count)
- print ("Elapsed Time: %s" % (time.time() - start))
- except OSError, e:
- print >>sys.stderr, "fork #1 failed: %d (%s)" % (e.errno, e.strerror)
- sys.exit(1)
- n = n + 1
- if __name__ == '__main__':
-
-
-
-
-
-
-
-
-
- try:
- loop = Loop(5, 100, 10000)
- except KeyboardInterrupt:
- print ("Crtl+C Pressed. Shutting down.")
- os.exit()
- $ cat table.py
-
- import random,string
- def <span style="background-color: #ffffff;">test0</span>():
- sql =
-
-
-
-
-
-
-
-
-
-
-
收藏
举报
TAG: