Go Back   TechArena Community > Software > Software Development
Become a Member!
Forgot your username/password?
Register Tags Active Topics RSS Search Mark Forums Read

Sponsored Links



Python urllib2 utf-8 and threading

Software Development


Reply
 
Thread Tools Search this Thread
  #1  
Old 18-05-2009
Member
 
Join Date: May 2008
Posts: 191
Python urllib2 utf-8 and threading
  

Hello,

I have some problems to understand the threading module and the few bits of code I found does not say more.

Here is my simple code of departure (it recursive html pages and sending a message if everything is ok):
Quote:
  1. #! / usr / bin / env python2.6
  2. # -*- Coding: utf-8 -*-
  3. import thread
  4. import urllib2
  5. class robot:
  6. def __init__ (self, url):
  7. url = self.__url
  8. def queue (self):
  9. for page in self.__url:
  10. html = self.execute (page)
  11. if html:
  12. self.sendmail (the content is ok)
  13. else:
  14. print "error on page (0)." format (page)
  15. def execute (self, url):
  16. "" "
  17. recursive html content ...
  18. "" "
  19. urllib2.url open return (url). read ()
  20. def sendmail (self, message):
  21. "" "
  22. sending a mail ...
  23. "" "
  24. pass
  25. if __name__ == '__main__':
  26. url = [ "http://google.com", "http://www.techarena.in"]
  27. recursive = robot (url)
  28. recursive.queue ()
It works but will cause speed problems (depending on the number of pages, or pages that are blocked: timeout ...). In short I paralleled the "execute" to use the threading module.

The worry is that I need to recover the results of this function to generate tests / messages and I do not see how ...

thank you to those who could enlighten me

Reply With Quote
  #2  
Old 18-05-2009
Member
 
Join Date: May 2008
Posts: 297
Re: Python urllib2 utf-8 error

Use (Queue.Queue) with the URLs to the entrance, a queue for output.
Reply With Quote
  #3  
Old 18-05-2009
Member
 
Join Date: May 2008
Posts: 191
Re: Python urllib2 utf-8 error

I used the multiprocessing module is more efficient for what I have to do.

I modified the function queue ():

Quote:
def queue (self):
time_start datetime.datetime.now = ()
processes = [code]
for page in self.__url:
p = Process (target = self.execute, args = (page))
p.start ()
processes.append (p)
for p in processes:
p.join ()
# For page in html:
# If page:
# Self.sendmail (the content is ok)
# Else:
# Print "error on page (0)"
time_end = datetime.datetime.now ()
print "execute in (0) sec." format (time_end - time_start)
This works but a few questions:
- A process does not return "nothing". It means execute my function which returns the html code, I could not recover. Me, I manage to integrate the test on the html code directly in the execute ()?
Reply With Quote
Reply

  TechArena Community > Software > Software Development
Tags:



Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads for: "Python urllib2 utf-8 and threading"
Thread Thread Starter Forum Replies Last Post
Threading in c# .net Athreya Software Development 3 12-01-2011 03:29 PM
What is Hyper threading Ameeryan Overclocking & Computer Modification 2 06-10-2009 05:03 PM
Python Threading - Basics Zecho Software Development 3 14-04-2009 03:00 PM
What is Threading in C# Programming ? HAKAN Software Development 2 31-03-2009 12:18 PM
Download Python 3.0 / Python 3000 Amaresh Software Development 6 24-02-2009 09:28 AM


All times are GMT +5.5. The time now is 09:48 AM.