Import Txt documents

1C:Enterprise platform integration capabilities and techniques

#1
People who like this:0Yes/0No
Just came
Rating: 0
Joined: Oct 31, 2011
Company: Smart ID Dynamics

I must import a large txt document with 256 columns and 200,000 rows. I use the TextReader method, but it last very  long. Can you help me with other idea of making the import so it work faster?

 
#2
People who like this:0Yes/0No
Just came
Rating: 0
Joined: Nov 1, 2011
Company:

If executioning context is Windows, then you can use MS FileSystemObject,

for example:
* bugt: removed by administrator. non-english sites links

Total line:

Code
    fso = NewCOMObject ("Scripting.FileSystemObject");
    file = fso.GetFile(stringFileName);
    
    TextStream = file.OpenAsTextStream(1, -1);
    //file = fso.OpenTextFile(stringFileName, 1, , 0);
 
    While file.AtEndOfStream=0 Do
        EachLineOfTextStream= TextStream.ReadAll();
        LetsHandlThisLine(EachLineOfTextStream);
    EndDo; 
 
    TextStream.Close();




Something like this

 
#3
People who like this:0Yes/0No
Just came
Rating: 0
Joined: Mar 12, 2012
Company:

I imported txt documents with 600,000 row and 55-60Mb size. I didn't have problem to read txt document. I used "TextDocument" method. I had another problem with the recording speed of the data into the database.
Do you write those data into the base ?

 
#4
People who like this:0Yes/0No
Just came
Rating: 1
Joined: Oct 29, 2011
Company: Smart ID Dynamics

In fact, the problem is saving/updating in the database, not reading the text file.
It is any posibility to bulk insert / update a catalog in the database with data from a text file?

 
#5
People who like this:0Yes/0No
Just came
Rating: 0
Joined: Nov 1, 2011
Company:

Actually it depends on many factors - first of all - what kind of data we want to ins ert in database. If it's InformationRegister - you can read whole text file, and insert it in database at once, in one RecordSet.
For AccumulationRegister may be more subtle to invoke Se tTotalsUsing(false) before writing data in database.

 
#6
People who like this:0Yes/0No
Just came
Rating: 1
Joined: Oct 29, 2011
Company: Smart ID Dynamics

The file is more a catalog than an InformationRegister. It is in csv format.
Because the file is very large and we get out of memory, we actually process it reading groups of lines, check if the records are already in catalog an update or insert if it is not there. But this takes very long time.
Any sugestion is verry wellcome.

 
#7
People who like this:0Yes/0No
Just came
Rating: 0
Joined: Nov 1, 2011
Company:

#6, what exactly takes long time - can you elaborate the whole process with more details?

 
#8
People who like this:0Yes/0No
Just came
Rating: 1
Joined: Mar 15, 2012
Company: no

Did you try the TextReader object?

It was designed for sequential read text files with long length.

Code
Text = New TextReader("d:\win.txt");

 
Subscribe
Users browsing this topic (guests: 1, registered: 0, hidden: 0)
Be the first to know tips & tricks on business application development!

A confirmation e-mail has been sent to the e-mail address you provided .

Click the link in the e-mail to confirm and activate the subscription.