Python urllib2 utf-8 and threading

[Yaropolk]

Hello,

I have some problems to understand the threading module and the few bits of code I found does not say more.

Here is my simple code of departure (it recursive html pages and sending a message if everything is ok):

1. #! / usr / bin / env python2.6
2. # -*- Coding: utf-8 -*-
3. import thread
4. import urllib2
5. class robot:
6. def __init__ (self, url):
7. url = self.__url
8. def queue (self):
9. for page in self.__url:
10. html = self.execute (page)
11. if html:
12. self.sendmail (the content is ok)
13. else:
14. print "error on page (0)." format (page)
15. def execute (self, url):
16. "" "
17. recursive html content ...
18. "" "
19. urllib2.url open return (url). read ()
20. def sendmail (self, message):
21. "" "
22. sending a mail ...
23. "" "
24. pass
25. if __name__ == '__main__':
26. url = [ "http://google.com", "http://www.techarena.in"]
27. recursive = robot (url)
28. recursive.queue ()

It works but will cause speed problems (depending on the number of pages, or pages that are blocked: timeout ...). In short I paralleled the "execute" to use the threading module.

The worry is that I need to recover the results of this function to generate tests / messages and I do not see how ...

thank you to those who could enlighten me

[SimonBoss]

Use (Queue.Queue) with the URLs to the entrance, a queue for output.

[Yaropolk]

I used the multiprocessing module is more efficient for what I have to do.

I modified the function queue ():

def queue (self):
time_start datetime.datetime.now = ()
processes = [code]
for page in self.__url:
p = Process (target = self.execute, args = (page))
p.start ()
processes.append (p)
for p in processes:
p.join ()
# For page in html:
# If page:
# Self.sendmail (the content is ok)
# Else:
# Print "error on page (0)"
time_end = datetime.datetime.now ()
print "execute in (0) sec." format (time_end - time_start)

This works but a few questions:
- A process does not return "nothing". It means execute my function which returns the html code, I could not recover. Me, I manage to integrate the test on the html code directly in the execute ()?