tarifa Posted March 30, 2020 Share Posted March 30, 2020 dear Fellow Coder on Neowin im using bash to pipe the gathered data through an Rscript like so: cat random.csv | Rscript test.R arg >| delete.csv My aim is to use the R package readr to both read stdin and write stdout. what is aimed: I found the answer to stdin here. test.R #!/usr/bin/Rscript suppressMessages(library(readr)) args <- commandArgs(trailingOnly = TRUE) df.in <- read_csv(file("stdin")) write_csv(df.in, path = stdout()) My investigations gave back the foolowing: There is a format_csv function for that in readr. Use this instead of write_csv: cat(format_csv(df.in)) we also can use write.table: write.table(x, file = "foo.csv", sep = ",", col.names = NA, qmethod = "double") i found the following interesting help page: https://stat.ethz.ch/R-manual/R-devel/library/utils/html/write.table.html import requests from bs4 import BeautifulSoup import pandas as pd def Main(urls): with requests.Session() as req: allin = [] for url in urls: r = req.get(url) soup = BeautifulSoup(r.content, 'html.parser') target = soup.find( "dl", class_="c-description-list c-description-list--striped") names = [item.text for item in target.findAll("dt")] names.append("url") data = [item.get_text(strip=True) for item in target.findAll("dd")] data.append(url) allin.append(data) df = pd.DataFrame(allin, columns=names) df.to_csv("data.csv", index=False, encoding="utf-8") urls = ['https://www2.daad.de/deutschland/studienangebote/international-programmes/en/detail/4722/', 'https://www2.daad.de/deutschland/studienangebote/international-programmes/en/detail/6318/'] Main(urls) the question is. i want to write the results of the parser script to stdout . - i do not want to write it into the data-file. Link to comment Share on other sites More sharing options...
0 +virtorio MVC Posted March 30, 2020 MVC Share Posted March 30, 2020 According to this: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html "to_csv()" returns a string if no file path/object is specified, so you should be able to print the result to stdout. Link to comment Share on other sites More sharing options...
Question
tarifa
dear Fellow Coder on Neowin
im using bash to pipe the gathered data through an Rscript like so:
cat random.csv | Rscript test.R arg >| delete.csv
My aim is to use the R package readr to both read stdin and write stdout. what is aimed: I found the answer to stdin here.
test.R #!/usr/bin/Rscript suppressMessages(library(readr)) args <- commandArgs(trailingOnly = TRUE) df.in <- read_csv(file("stdin")) write_csv(df.in, path = stdout())
My investigations gave back the foolowing:
There is a format_csv function for that in readr. Use this instead of write_csv:
cat(format_csv(df.in))
we also can use write.table:
write.table(x, file = "foo.csv", sep = ",", col.names = NA, qmethod = "double")
i found the following interesting help page: https://stat.ethz.ch/R-manual/R-devel/library/utils/html/write.table.html
import requests from bs4 import BeautifulSoup import pandas as pd def Main(urls): with requests.Session() as req: allin = [] for url in urls: r = req.get(url) soup = BeautifulSoup(r.content, 'html.parser') target = soup.find( "dl", class_="c-description-list c-description-list--striped") names = [item.text for item in target.findAll("dt")] names.append("url") data = [item.get_text(strip=True) for item in target.findAll("dd")] data.append(url) allin.append(data) df = pd.DataFrame(allin, columns=names) df.to_csv("data.csv", index=False, encoding="utf-8") urls = ['https://www2.daad.de/deutschland/studienangebote/international-programmes/en/detail/4722/', 'https://www2.daad.de/deutschland/studienangebote/international-programmes/en/detail/6318/'] Main(urls)
the question is. i want to write the results of the parser script to stdout . - i do not want to write it into the data-file.
Link to comment
Share on other sites
1 answer to this question
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now