{"id":999,"date":"2011-01-19T20:41:57","date_gmt":"2011-01-20T01:41:57","guid":{"rendered":"http:\/\/appcrawler.com\/wordpress\/?p=999"},"modified":"2011-07-06T09:43:43","modified_gmt":"2011-07-06T14:43:43","slug":"python-multiple-producerconsumer-queue","status":"publish","type":"post","link":"http:\/\/appcrawler.com\/wordpress\/2011\/01\/19\/python-multiple-producerconsumer-queue\/","title":{"rendered":"Python multiple producer\/consumer queue"},"content":{"rendered":"<p>Below is a simple skeleton for a python queue that is both fed by and consumed by multiple different threads.<\/p>\n<p>We run five threads, each of which populate a queue with random integers.  <\/p>\n<p>Lastly, we sleep a couple of seconds between each put, to simulate an ongoing workload.  <\/p>\n<p>Our popper thread, which reads items off the queue, will wait for an item in the queue for ten seconds before throwing an Queue Empty exception.  We also print the thread name to prove each thread is doing its fair share of the workload.<br \/>\nIt will then loop until all the pusher threads have exited (as defined by our running count variable).<\/p>\n<pre lang=\"python\" line=\"1\">\r\nimport Queue, random, time, thread, sys\r\nfrom threading import Thread\r\n\r\nplock = thread.allocate_lock()\r\nrlock = thread.allocate_lock()\r\n\r\nq = Queue.Queue()\r\nrunning = 0\r\n\r\n#-----------------------------------------\r\n\r\nclass pusher(Thread):\r\n  def __init__(self):\r\n    global q\r\n    self.num = i\r\n    Thread.__init__(self)\r\n    self.q = q\r\n  def run(self):\r\n    global running\r\n    rlock.acquire()\r\n    running = running + 1\r\n    rlock.release()\r\n    for i in range(3):\r\n      l = random.randint(1,100)\r\n      plock.acquire()\r\n      print \"pusher \" + str(l) + \" \" + self.getName()\r\n      plock.release()\r\n      self.q.put(l);\r\n      time.sleep(2)\r\n    rlock.acquire()\r\n    running = running - 1\r\n    rlock.release()\r\n\r\n#-----------------------------------------\r\n\r\nclass popper(Thread):\r\n  def __init__(self):\r\n    global q\r\n    self.num = i\r\n    Thread.__init__(self)\r\n    self.q = q\r\n  def run(self):\r\n    global running\r\n    while running > 0:\r\n      try:\r\n        l = self.q.get(True,10);\r\n        plock.acquire()\r\n        print \"popper \" + str(l) + \" \" + self.getName()\r\n        plock.release()\r\n      except:\r\n        print str(sys.exc_info()[1])\r\n\r\n#-----------------------------------------\r\n\r\nfor i in range(5):\r\n  c = pusher()\r\n  c.start()\r\n\r\ntime.sleep(2)\r\n\r\nfor i in range(5):\r\n  c = popper()\r\n  c.start()\r\n <\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Below is a simple skeleton for a python queue that is both fed by and consumed by multiple different threads. We run five threads, each of which populate a queue with random integers. Lastly, we sleep a couple of seconds&hellip;<\/p>\n<p class=\"more-link-p\"><a class=\"more-link\" href=\"http:\/\/appcrawler.com\/wordpress\/2011\/01\/19\/python-multiple-producerconsumer-queue\/\">Read more &rarr;<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"footnotes":""},"categories":[24,26],"tags":[],"_links":{"self":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/999"}],"collection":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/comments?post=999"}],"version-history":[{"count":16,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/999\/revisions"}],"predecessor-version":[{"id":1288,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/posts\/999\/revisions\/1288"}],"wp:attachment":[{"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/media?parent=999"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/categories?post=999"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/appcrawler.com\/wordpress\/wp-json\/wp\/v2\/tags?post=999"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}