Efficient memory usage with iterators in Python
I'm completely new to Python. I have recently read the below three points about iterators:
1. An iterator is "non-destructive", ie you can not change the data structure as it loops through the iterator itself. They can therefore be used to make data "read only". Another way of saying this is: An iterator is "one way", data can be retrieved, but not submitted.
2. An iterator has consciousness - ie, it knows and remembers where the last in the sequence was. You do not need to keep track of an index.
3. An iterator is often memory-efficient. A typical example is the huge files. Let's say you have a log file on 1GB. The code to read this file will eat up your memory. But by looping over the file object via the built-in iterator it will never use more memory than every single line.
Coming to the point: Here is the code I'm working with:
Code:
# Open the webdocument with The First 10,000 Primes, read the file and print the content. It functions for Python version 2.5, But Not for version 3.1.2
# import urllib
file = urllib . urlopen ( 'http://primes.utm.edu/lists/small/10000.txt' )
primes = file . read ()
print ( primes )
Code:
# Brute force prime number and prime number calculation twin. There ar no security net, exception
# action in the code. Note That all twin prime except (3, 5) are of the form: (nx6) + / - 1
print('This prgram is only valid for the natural number Between 1 and 1700')
print()
primed= [3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59,61,67,71,73,79,83,89,97,101] # Python list inbuilt
import math # import math module
n = int(input ('Enter a natural number Between 1 and 1700:'))
m1 = n*6 - 1
m2 = n*6 + 1
print()
limit=int(math.ceil(math.sqrt(m2))) + 1 # Ceiling of square root of number + 1
bool1=1
for the prime in prime: # Loop through the list of known prime.
if prime <= limit:
if m1 % prime == 0: # Enough two test for prime factors up to the square root
bool1=0 # No prime
break # Break out of the loop if a prime factor is found.
if bool1 == 1:
print (m1, 'is prime')
else:
print (m1, 'is NOT prime')
print()
bool2=1
for the prime in prime: # Loop through the list of known prime.
if prime <= limit:
if m2 % prime == 0: # Enough two test for prime factors up to the square root
bool2=0 # No prime
break # Break out of the loop if a prime factor is found.
if bool2 == 1:
print (m2, 'is prime')
else:
print (m2, 'is NOT prime')
print()
print()
Enter=input('Hit enter') # Prevent output window from closing.
Problems:
- The first code works with Python 2.5, but not with Python 3.*. Maybe I'm missing some external library since Python 2.5 was installed from a third party with many external extension and a library.
- Switch the most effective method and iterate over the file object as suggested in the introduction.
- How to combine the first code and the second code so well-known prime number is read from the specified or custom website. Filter out the text in the preamble to the file on the website.
- Calculate even larger prime numbers that are saved to text/html file.
Re: Efficient memory usage with iterators in Python
Yes, reading a log file or any other file will take its content to the memory. It will take up all of your memory because Python uses all the available memory. So it is recommended to read line by line only into the memory. You can even use "with" statement to determine where the problem is occurring. It serves the error handling feature in Python just like the try catch in JAVA. About your problem points:
1. Its because urllib.urlopen () has been removed in python 3.x. It has been replaced with urllib.request.urlopen.
3.
Code:
import urllib.request
file = urllib.request.urlopen('http://primes.utm.edu/lists/small/10000.txt')
primes = file.read().decode("utf8")
my_primes = []
for i,line in enumerate(primes.split('\n')):
if i > 3 and i < 1004:
temp = ((int(item) for item in line.split()))
for i in temp:
my_primes.append(i)
print (my_primes)
#You can take out number you want.
#10 first prime number
print (my_primes[0:10])
Re: Efficient memory usage with iterators in Python
I do not know your needs, but there is a way to change the storage time in the case of prime numbers. Miller-Rabin is a very well known probabilistic algorithm for determining whether a given number is likely to be a prime number. For numbers <2 ^ 32, there is a special edition of the Miller-Rabin who always gives the correct answer and only 3 test candidates. you have a fast algorithm that is easy to implement and uses constant space (was quite useful in project-related formulations).
Re: Efficient memory usage with iterators in Python
Thanks for your useful input. I do this:
- First and foremost to learn Python.
- And while learning more about the subject. I know some of these algorithms. Now it is not at this stage the greatest importance to use the most effective algorithm, though no new programming challenges arise.
- But by all means, please come with suggestions for better algorithms. Everyone learns from it.
- Learning Python and Django for an alternative tool for web development with PHP.
- Could combine Python and C/C++. That is how I understand it's a good combination.
- etc.
Have barely skimmed the contents of some books and since I can now solve simple tasks in Python, I see that there is a very good choice as a first language learning and teaching to learn programming. Python is short, as I see it, near a pseudo language. Another issue that I probably can figure it out, but you can obviously learn in a few hours and days more than me. Have read something about "Generator expressions" that is new to me. Have understood that it can make programs more efficient, so what is it and can be used in the prime issue that is mentioned here? And it is easy to integrate PyGames and NumPy for Python 3 *? Both for version 2.5, so is there compatibility issues with the latest version of Python that I intend to use in any projects. Finally, since he who made the Python now works for Google, probably with their own languages go, I wonder how easy it is compared to Python.
Re: Efficient memory usage with iterators in Python
This is the fast prime number generator that I came across when I tested this a while ago. This is a good time 100000000 prime numbers down to C/C++ speeds.
Code:
import math
import time
import psyco
psyco.full()
start = time.clock()
def primCal(end):
if end < 2: return []
lng = ((end/2)-1+end%2)
primCount = [True]*(lng+1)
for i in range(int(math.sqrt(end)) >> 1):
if not primCount [i]: continue
for j in range( (i*(i + 3) << 1) + 3, lng, (i << 1) + 3):
primCount [j] = False
primes = [2]
primes.extend([(i << 1) + 3 for i in range(lng) if primCount [i]])
pri = primes[-1]
return pri
if __name__ == '__main__':
prime_nr = 100000
print primCal(prime_nr)
end = time.clock()
print '%s prime number in %.3f seconds' % (prime_nr, end - start)
'''
My output-->
99999989
100000000 prime_numbers with python in 7.923 seconds
'''
