Importing and cleaning several text files in R

Question

I’m working on a set of interviews I got from a database in my work, each in a separate text file (a normal file with a multi-line transcription).

Each interview will be part of data frame as a variable. The metadata of each interview is in a different data frame. I have to import the texts for him (one whole interview per line). There are more than 700 text files to be imported.

The issue is that there are many repeated lines in the original interview files and, due to the number of files, import one by one and use x <- unique (x) is out of the question. I wrote the code below to try to import them:

files <- list.files(path="path/to/the/files", pattern="*.txt", full.names=TRUE)
folder <- "path/to/the/folder"    
clean_text <- function(myfile){
  n <- length(myfile)
  myfile <- c(stringr::str_remove_all(myfile[-n], "[\n].*"),
              myfile[n])
  myfile <- unique(myfile)
  myfile <- paste0(myfile, collapse = "\n")
  myfile
}
texts <- ldply(clean_text(files),  read_file)

But instead of having something like this:

All I have is the path to my files

View(texts)
V1
path/to/my/files/file1 path/to/my/files/file2

I’m open to any solution.

Updating

Here’s a workflow of what I’ve been unable to do:

1. Read 700 text files from a directory (these are files from plain text with lines separated by n)
2. Delete the lines repeated
3. Import them to R. It can be a vector of a column in the which each text is a variable.

Update 2

I managed to evolve until I had the text in a list, where each one is a Character:

 str(teste)
List of 3
 $ : chr [1:205] "Hello " ...
 $ : chr [1:581] "hello little buns  ...
 $ : chr [1:849] "- Hello everybody," ...

But I can’t export each of these elements from the list within a single cell of a data frame. I’ve already reviewed Google.

If anyone could help, I’d be grateful.

Answer 1

If I understood the question (I doubt you have) maybe the following code can solve the problem.
The function below uses the Unix/linux command awk to remove duplicate lines of text. Clean files are written to files with the same name as the input files but prefixed with the pref = "out" in their names.

clean_text2 <- function(file, pref = "out"){
  out <- file.path(dirname(file), paste0(pref, basename(file)))
  args <- c("'!_[$0]++'", file, ">", out)
  system2("awk", args)
}

lapply(files, clean_text2)

Answer 2

Thank you all. Apparently I was able to solve the problem with the following function. I did a test on a sample of 10 files and it worked. Then I test with everyone and give feedback.

files <- list.files(path="path/to/you/files", pattern="*.txt", full.names=TRUE)
import.and.clean <- function(myfiles){
  tmp <- lapply(myfiles, read_lines)
  tmp2 <- purrr::map(tmp, unique)
  tmp3 <- purrr::map_chr(tmp2, paste0, collapse = "\n")
  return(tmp3)
}
my.vector <- import.and.clean(files)