Rating:
0
Joined: Oct 31, 2011
Company: Smart ID Dynamics
I must import a large txt document with 256 columns and 200,000 rows. I use the TextReader method, but it last very long. Can you help me with other idea of making the import so it work faster?
I imported txt documents with 600,000 row and 55-60Mb size. I didn't have problem to read txt document. I used "TextDocument" method. I had another problem with the recording speed of the data into the database. Do you write those data into the base ?
Rating:
1
Joined: Oct 29, 2011
Company: Smart ID Dynamics
In fact, the problem is saving/updating in the database, not reading the text file. It is any posibility to bulk insert / update a catalog in the database with data from a text file?
Actually it depends on many factors - first of all - what kind of data we want to ins ert in database. If it's InformationRegister - you can read whole text file, and insert it in database at once, in one RecordSet. For AccumulationRegister may be more subtle to invoke Se tTotalsUsing(false) before writing data in database.
Rating:
1
Joined: Oct 29, 2011
Company: Smart ID Dynamics
The file is more a catalog than an InformationRegister. It is in csv format. Because the file is very large and we get out of memory, we actually process it reading groups of lines, check if the records are already in catalog an update or insert if it is not there. But this takes very long time. Any sugestion is verry wellcome.