My script looking for a string in another string is taking too long

Question

My script loads the command1(file1) and saves the first column of all rows in an array, later it will search for each substring of that array in a 2(from command2) file. The problem is that file 1 has about 3,000,000,000 lines and file 2 has about 25,000,000 lines. Soon I needed a program that was as fast as possible.

Ex file 1:

sp|Q8EES8|Y2290_SHEON   G0B6XZL01A4P7W_6    67.77   121 39  0   128 248 1   121 3e-06    188

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01ARMHK_3    57.58   132 56  0   169 300 3   134 1e-06    180

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01B0A9N_1    47.37   152 72  3   124 269 1   150 5e-06    150

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01AS06A_2    51.40   107 52  0   173 279 46  152 9e-03    136

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01BI3RW_5    41.10   146 85  1   50  194 10  155 3e-03    129

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01DQILJ_4    45.95   111 60  0   176 286 46  156 1e-02    117

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01ATWAG_1    35.26   156 95  2   19  173 2   152 5e-02    110

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01AYTV2_4    37.88   132 80  2   83  212 32  163 7e-02    102

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01C7I53_6    44.12   102 56  1   112 212 1   102 2e-02   92.4

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01B9TOA_5    42.98   114 64  1   4   117 41  153 1e-01   86.3

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01DQILJ_5    54.93   71  29  2   129 199 1   68  5e-01   84.7

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01E13OT_2    38.10   105 65  0   1   105 6   110 6e-01   84.0

sp|Q8EES8|Y2290_SHEON   G0B6XZL01EGX3B_4    33.56   149 91  4   46  189 1   146 4e-01   79.0

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01EEMHX_3    40.48   84  50  0   173 256 31  114 7e-01   78.6

sp|Q8EES8|Y2290_SHEON   G0B6XZL01BDBAI_3    52.83   53  25  0   241 293 2   54  3e-01   74.7

sp|Q8EES8|Y2290_SHEON   G0B6XZL01ETJ9Y_6    51.67   60  29  0   242 301 1   60  3e-01   75.5

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01ARVZB_6    43.04   79  44  1   216 293 8   86  9e-01   70.5

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01EFORR_1    54.55   55  25  0   219 273 3   57  1e-01   66.6

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01DWDKC_1    47.27   55  29  0   219 273 5   59  1e-01   66.6

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01AL4M3_1    47.27   55  29  0   219 273 5   59  3e-01   65.5

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01B16CL_2    45.83   48  26  0   111 158 66  113 2e-03   57.4

sp|Q8EES8|Y2290_SHEON   G0B6XZL01D8VWQ_6    37.18   78  49  0   169 246 6   83  4e-03   55.5

sp|Q8EES8|Y2290_SHEON   G0B6XZL01D8VWQ_5    61.11   36  14  0   176 211 13  48  5e-03   55.5

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01EN153_2    39.13   69  40  2   104 171 88  155 1e-02   55.5

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01DJ7AX_5    32.65   98  65  1   4   100 96  193 2e-04   55.5

sp|Q8EES8|Y2290_SHEON   GRQ41VZ01AG6GR_1    27.04   159 94  6   5   147 19  171 3e-04   53.9

Ex file 2:

">G0B6XZL01A4P7W_6 length=363 xy=0346_3114 region=1 run=R_2011_04_01_12_14_13_
LVIDTRNEYEVEIGTFAGAVNPHTNSFREFPDWVEQNLDPKKHKKVAXFCTGGIRCEKST
SLLVSRGFEDVWHLKGGILNYLEQTPEEDTRWEGECFVFDSRVAVNHQLEKGSYDQCFAC
R"

">GRQ41VZ01ARMHK_3 length=428 xy=0197_2054 region=1 run=R_2010_11_26_14_54_57_
DKKKRKLQVFCTGGIRCEKASSLMKKEGFENVYHLKGGILKYFESVNEDDSLWSGECFVF
DDRVSVDQNLEKGSYDMCHGCRMPITINDKKTDKYIRGVACPSCFDKTTEEQKNRYMSRQ
KQVDLAKKKKYKNILGPKKRSY"

">GRQ41VZ01B0A9N_1 length=466 xy=0706_2169 region=1 run=R_2010_11_26_14_54_57_
DPDTLVIDTRNSYETAIGSFEGAIDPSTESFRDFPQWAESTLRPLIEEKGSKRIAMFCTG
GIRCEKASSYLQQQGFGEVHHLRGGILKYFEQVPEAESRWQGECFVLINGWR*TTGWNLE
STAFATPAACRCQPSNANCRATSKGVQCVHVRGSLX"

">GRQ41VZ01BWIB5_5 length=457 xy=0663_0835 region=1 run=R_2010_11_26_14_54_57_
RHPHIKDKVPQ*MFHPELFDALSESFVELSLFVSVHLVFRRSSTCLRVLDIFCFSRSECL
YPCLP*NIK*P*IIIVIPKRNNITTPN*SINVDEAMALLVE*PQ*VLTDPH*RVQEQRNQ
QTY**FQHFDYAFLWHHRSSPDTLHE*HNALVX"

">GRQ41VZ01BWIB5_3 length=457 xy=0663_0835 region=1 run=R_2010_11_26_14_54_57_
TSALCHS*RVSGDDR*CHKKA*SKCWNY*YVCWFLCSWTL**GSVKTYWGYSTSSAMASS
TFIDQLGVVMLFLFGITMIIQGHFIFHGKHGYKHSEREKQKMSKTRKQVEDLLKTK*TLT
KRESSTKDSESASNNSGWNIY*GTLSFI*G*R"

">GRQ41VZ01BDBOS_6 length=518 xy=0444_2778 region=1 run=R_2010_11_26_14_54_57_
VVNLVDT*KTLMLYLKAQRNYKAS*KNSCIVW*SYFI*SSY*NRRLS*CRFSSIYLGGRY
*RRVQKYRYVI**TIQDEKIAYATYCYNKNR*A*R*HWFILCS*LFTR*EK*YSYN***P
KYL*RIYGIHR*T*YLGFY*VG*K**RRNKLSYCQRIY*YRCYAIILLKHTIX"

Code:

palavras=[]
paragrafo=[]
palavraslinha=[]
x=[]
z=[]
i=0
j=0
k=[]        
q=[]        
w=[]        
e=['>']
t=int(0)
f=0
p= int (0)
fasta='.fasta'
print('Todos os arquivos devem estar no mesmo diretório. Basta digitar o                         
nome do arquivo que o formato ".fasta" será incluído automaticamente.')
comando1=input('Digite o nome do arquivo de parâmetro: ')
comando2=input('Digite o nome do arquivo de busca: ')
comando3=input('Digite o nome do arquivo a ser criado: ')
paragrafo_fim=[]
ref_arquivo1 = open(comando2+fasta,"r")
linha = ref_arquivo1.read()
with open(comando1+fasta) as f:
    for line in f:
        x.append(line.split())
        z=x[i][1]
        k.append(z)

        i+=1
    x=[]
    z=[]
while linha:
    palavras.append(linha.split('>'))
    while t<len(k):
        w.append(k[t])
        while p<len(palavras[0]):
            if w[t] in palavras[0][p]:
                arquivofim = open(comando3+fasta,'a')
                arquivofim.write(e[0]+palavras[0][p])
                del palavras[0][p]
                p+=1 
                #print (w[t])
                arquivofim.close()
            else:
                p+=1
        print(t)
        t= t+1
        p=0
    linha = ref_arquivo1.readline()
print('O processamento terminou')    
ref_arquivo1.close()

Answer 1

From what I understand you are making a filter. If so, the problem is that you are putting too much in memory without need. Python is no better for that, then search for an ETL like Kettle https://community.hitachivantara.com/docs/DOC-1009855-data-integration-kettle

If I understood correctly one wiped in your code would look like this:

fasta='.fasta'
print('Todos os arquivos devem estar no mesmo diretório. Basta digitar o nome do arquivo que o formato ".fasta" será incluído automaticamente.')
comando1=input('Digite o nome do arquivo de parâmetro: ')
comando2=input('Digite o nome do arquivo de busca: ')
comando3=input('Digite o nome do arquivo a ser criado: ')


novo_f = open(comando3 + fasta, 'a');

# gera uma lista com o que importa pegar
chaves = []
with open(comando1+fasta) as f:
    for linha in f:
        chaves.append(linha.split(maxsplit=2)[1])

salvar = False
with open(comando2+fasta,"r") as f:
    for linha in f:
        # verifica se é um novo bloco de linhas
        if linha.startswith('"'):
            # verifica se o valor está na lista, removendo todo o resto
            # que é desnecessário. Se for, então começa a gravar
            if linha[2:linha.find(' ')] in chaves:
                salvar = True

        if salvar:
            novo_f.write(linha)

            # finaliza o bloco de linhas
            if linha.endswith('"\n') or linha.endswith('"'):
                salvar = False

novo_f.close()