| || |
Join Date: Apr 2012
Need C# script to Read and Write Lines over large database
Now there’s something interesting I would like to share and learn from all my programming friends. I am a software developer specialized in C++ language. Now I want to ask you all something interesting, can anyone tell me the possible way of reading and writing a 500 GB of data using this C++ program. Its better I would explain you in detail, there are about 500 text files existing into my system and the overall size of those files altogether is about 500GB. As far as I know the largest file consists of more than 7 GB.
I need a kind of program that could simply read each line, each time from each file and then manipulate it as according to my requirement. Now the most interesting thing is that I all of my 500 files contains exactly 81 characters and a CRLF in each line of each file. As I said I want the program to read all lines individually then manipulate them but the thing is that those 81 characters shouldn’t be changed or modified, they should be maintained as they are. Once done with that then the manipulated line should be written to some other text file. Its not that I haven’t tried or researched anything regarding what I am wanting. Already I have found a program on web that do the same task but there are certain differences and drawbacks that do not match my criteria at all. The main issue with that program is that it simply reads out the line and writes it on another file but doesn’t do it accurately and efficiently which is very much important for me. However after implementing and using the programs I found that are there lots of trouble and issues that exist within it. When I tested, I found that even for reading a smaller size file the program used to take hours which is not at all expected and also after the completion of the reading process I noticed that the process ended over there. I mean to say that there was no further process like writing took place in the program. Is there anyway to make this program perfect and run it as according to our requirement. If needed I would definitely provide you with the coding of this improper program.
Regarding my system specification: I am running windows 7 64bit into my machine with 4GB RAM and there’s no such seriously issue or problem within the system. Hope any changes suggested by you might be according to my current system specification.