EmEditor (text editor) Forum Index
   Regular Expressions
     how to delete duplicate lines
Register To Post

Threaded | Oldest First Previous Topic | Next Topic | Bottom
Poster Thread
Hellados
Posted on: 4/18/2009 1:10 am
Just popping in
Joined: 4/18/2009
From:
Posts: 1
Re: how to delete duplicate lines
This is a good macros for small files, but it is very slow for me
I have more 50-100mb txt files, and i need to replace dublicate lines (words) more then 406000 words and this macro working very slow :(
my pc's performances is very good, I have Intel COre 2 Duo E8400 2GB ram corsair 1TB HDD
What can i do?
user
Posted on: 9/29/2006 12:42 pm
Home away from home
Joined: 9/29/2006
From:
Posts: 212
Re: how to delete duplicate lines
thank you for your reply

my congrats and best wishes with the new forums :)
Yutaka
Posted on: 9/29/2006 11:56 am
Webmaster
Joined: 9/28/2006
From: Redmond
Posts: 2424
Re: how to delete duplicate lines
It isn't easy to do with regular expressions, but how about a macro like this (JavaScript):


// Create an array
a = new Array();

// Fill the array a with all lines (with returns) in the document.
document.selection.StartOfDocument();
for( ; ; ){
    y = document.selection.GetActivePointY( eePosLogical );
    document.selection.SelectLine();
    sLine = document.selection.Text;
    if( sLine == "" ) {   // Reached the end of document, escape from the loop
        break;
    }
    a.push( sLine );
    document.selection.Collapse();
    if( document.selection.GetActivePointY( eePosLogical ) == y ) {
        // Reached the end of document (the last line without return), escape from the loop
        break;
    }
}

// Delete duplicate elements.
for( i = 0; i < a.length; i++ ){
    sLine = a[i];
    for( j = i + 1; j < a.length; j++ ){
        if( sLine == a[j] ){
            a.splice( j, 1 );
            j--;
        }
    }
}

// Replace the entire document with new elements
document.selection.SelectAll();
document.selection.Text = a.join( "" );


Please let me know if you have questions.


----------------
Yutaka Emura
Developer of EmEditor
http://www.emeditor.com/

user
Posted on: 9/29/2006 7:27 am
Home away from home
Joined: 9/29/2006
From:
Posts: 212
how to delete duplicate lines
hello

how do I delete duplicate lines?

I mean lines that are identical

thanks
« 1 (2)
Threaded | Oldest First Previous Topic | Next Topic | Top


Register To Post
 
English čeština Deutsch español français italiano 日本語 한국어 Русский 简体中文 繁體中文