Need C# script to Read and Write Lines over large database

[Budhil]

Sponsored Links

Now there’s something interesting I would like to share and learn from all my programming friends. I am a software developer specialized in C++ language. Now I want to ask you all something interesting, can anyone tell me the possible way of reading and writing a 500 GB of data using this C++ program. Its better I would explain you in detail, there are about 500 text files existing into my system and the overall size of those files altogether is about 500GB. As far as I know the largest file consists of more than 7 GB.

I need a kind of program that could simply read each line, each time from each file and then manipulate it as according to my requirement. Now the most interesting thing is that I all of my 500 files contains exactly 81 characters and a CRLF in each line of each file. As I said I want the program to read all lines individually then manipulate them but the thing is that those 81 characters shouldn’t be changed or modified, they should be maintained as they are. Once done with that then the manipulated line should be written to some other text file. Its not that I haven’t tried or researched anything regarding what I am wanting. Already I have found a program on web that do the same task but there are certain differences and drawbacks that do not match my criteria at all. The main issue with that program is that it simply reads out the line and writes it on another file but doesn’t do it accurately and efficiently which is very much important for me. However after implementing and using the programs I found that are there lots of trouble and issues that exist within it. When I tested, I found that even for reading a smaller size file the program used to take hours which is not at all expected and also after the completion of the reading process I noticed that the process ended over there. I mean to say that there was no further process like writing took place in the program. Is there anyway to make this program perfect and run it as according to our requirement. If needed I would definitely provide you with the coding of this improper program.

Regarding my system specification: I am running windows 7 64bit into my machine with 4GB RAM and there’s no such seriously issue or problem within the system. Hope any changes suggested by you might be according to my current system specification.

[AabheerM]

Hi, I am also a C++ programmer like you and also a part time teacher for the same. Anyways there’s not any requirement of providing the entire code or program for any reference as I am well aware of the logics and the coding for the same. I know there does only one such program exist on internet and even I have it in my system which I edited by myself. Now what I would like to suggest you is modify the program in such a way where you can simply read block of data rather than reading each line at a time. I personally prefer using 4096-byte blocks but unfortunately the best suggested thing is that every block is actually divisible by 512 bytes which I guess depends upon those individual sectors on the disk.

Another thing I would like to let you do is to make use of less of memory as much as possible because as a programmer you must know that access utilization of memory ultimately results in slow running of the program and therefore hampers the overall performance of the same. Maybe such things happen just because of the virtual memory.

[Fred Fruax]

Ok, I am not sure whether you will like my suggestion but definitely it might help you in the best way possible. As already suggested above having to read just single line would really make your work too slow. I guess you even need an output and as far as I know even output takes bit time to get executed. Now just imagine your overall execution with such slow process and slow result. Make sure that you try to create a file name array just before you create a str() method. Another suggestion I would like to give you is to at least make two to three threads in-between your program. This might somehow help you to increase the speed of your execution. You can manage to make one for the data reading purpose then the second one for writing the data and if in case you have the third one then you keep it for the processing. You can take some help from someone who is very well aware of thread concept and ask him whether the above suggested thing is possible or not.