Read XML Data with rvest using R - GeeksforGeeks (2024)

XML stands for Extensible Markup Language, XML is just wrapping information in tags. It is a widely used markup language designed for storing and transporting data in a structured and human-readable format. It is designed to carry data and only focus on “what data is”. It is a text-based format that uses tags to define the structure and content of data.

Key Features of XML Data

• In XML each element can contain other elements, attributes, and text and gives a hierarchically tree-like structure with nested elements.
• XML has no predefined tags and the tags XML uses to define data elements is invented by the author. Tags are enclosed in angle brackets(‘< >’). Example: ‘<title>‘ is a tag that might define a title element in XML.
• Attributes are words containing a key and its value that provide additional information about the element. For example: ‘<movie genre=”Sci-Fi”>’.
• XML data is self-describing, meaning that the tags used to encapsulate data also describe the data’s meaning.
• XML allows users to define their own tags and data structures. This flexibility makes XML suitable for a wide range of applications.
• XML is platform-independent, it is easier to exchange data between different systems and technologies.

<library>
 <book>
 <title>The God of Small Things</title>
 <author>Arundhati Royr</author>
 <year>1997</year>
 <genre>Fiction</genre>
 </book>
 <book>
 <title>Gitanjali</title>
 <author>Rabindranath Tagore</author>
 <year>1910</year>
 <genre>Poetry</genre>
 </book>
</library>

Let’s breakdown the above example:

• <library>: The root element that contains all the entries of the book
• <book> : entry of each book is within the <book> tag.
• <title>, <author>, <year>, <genre>: these are the child elements or tags within the main <book> element, containing specific information about the book.

XML is used for Data Storage, Data Exchange, and Document Formatting.

What is rvest?

It is the most popular library for web Scraping from any public web page in R. rvest is an R package designed for web scraping, making it easy to extract data from web pages. It makes it simpler and simpler to work with HTML and XML documents by providing a set of the functions to navigate and parse these structures. So, the rvest library lets users easily scrape(“harvest”) data from web pages. It is one of the tidyverselibraries, so works well along with the other libraries, contained in the bundle.

Key Features of rvest:

• rvest allows to easily read HTML and XML documents into R for further analysis also known as parsing HTML/ XML.
• we can select specific elements from web pages with the help of CSS selectors and XPath.
• Extract text, attributes, and other content from the elements.
• seamlessly Integration with ‘tidyverse’ for data manipulation and cleaning.

Now we will create one XML File to Read XML Data with rvest in R Programming Language.

Namespaces are used in XML to avoid naming conflicts by differentiating elements and attributes that may have the same name but come from different sources or domains. This may can complicate data extraction. For example, two different XML schemas might use an element called <title>, but one could refer to a book’s title, and the other to a job title.

<languages xmlns:lang="http://example.org/language"> <lang:language id="1" difficulty="moderate"> <lang:name>Python</lang:name> <lang:type>Dynamic</lang:type> <lang:first_appeared>1991</lang:first_appeared> <lang:paradigm>Object-oriented, Imperative, Functional</lang:paradigm> </lang:language> <lang:language id="2" difficulty="hard"> <lang:name>Java</lang:name> <lang:type>Static</lang:type> <lang:first_appeared>1995</lang:first_appeared> <lang:paradigm>Object-oriented, Imperative</lang:paradigm> </lang:language> <lang:language id="3" difficulty="easy"> <lang:name>JavaScript</lang:name> <lang:type>Dynamic</lang:type> <lang:first_appeared>1995</lang:first_appeared> <lang:paradigm>Event-driven, Functional, Imperative</lang:paradigm> </lang:language></languages>

Now we will save this file and upload in our R Studio and start Read XML Data with rvest.

# Load the necessary librarieslibrary(rvest)library(xml2)library(dplyr)library(tidyr) # This provides the unnest() function# Parse the XML dataxml_data <- read_xml("path/to/your/directory/codelanguage.xml") # or you can add url as well# Identify namespaces in the XMLnamespaces <- xml_ns(xml_data)# Extract language nameslanguage_names <- xml_data %>% xml_find_all(".//lang:name",ns= namespaces) %>% xml_text()# Extract the id attribute from all <language> elementslanguage_ids <- xml_data %>% xml_find_all(".//lang:language", ns = namespaces) %>% xml_attr("id")# Extract the difficulty attribute from all <language> elementslanguage_difficulties <- xml_data %>% xml_find_all(".//lang:language", ns = namespaces) %>% xml_attr("difficulty")# Extract the type of languageslanguage_types <- xml_data %>% xml_find_all(".//lang:type", ns = namespaces) %>% xml_text()# Extract the first appeared year of languagesfirst_appeared <- xml_data %>% xml_find_all(".//lang:first_appeared", ns = namespaces) %>% xml_text()# Extract language paradigmslanguage_paradigms <- xml_data %>% xml_find_all(".//lang:paradigm", ns = namespaces) %>% xml_text()# Combine all extracted data into a dataframelanguages_df <- data.frame( ID = language_ids, Name = language_names, Type = language_types, Difficulty = language_difficulties, First_Appeared = first_appeared, Paradigm = language_paradigms, stringsAsFactors = FALSE)# Clean and transform the data using dplyr and tidyrlanguages_cleaned <- languages_df %>% mutate( ID = as.integer(ID),  First_Appeared = as.integer(First_Appeared),  Difficulty = factor(Difficulty, levels = c("easy", "moderate", "hard")),  Paradigm = strsplit(Paradigm, ", ")  ) %>% unnest(cols = c(Paradigm)) %>% arrange(First_Appeared) # Print the cleaned and transformed dataframeprint(languages_cleaned)

Output:

# A tibble: 8 × 6
 ID Name Type Difficulty First_Appeared Paradigm 
 <int> <chr> <chr> <fct> <int> <chr> 
1 1 Python Dynamic moderate 1991 Object-oriented
2 1 Python Dynamic moderate 1991 Imperative 
3 1 Python Dynamic moderate 1991 Functional 
4 2 Java Static hard 1995 Object-oriented
5 2 Java Static hard 1995 Imperative 
6 3 JavaScript Dynamic easy 1995 Event-driven 
7 3 JavaScript Dynamic easy 1995 Functional 
8 3 JavaScript Dynamic easy 1995 Imperative 

let’s understand what the above code is doing:

• Loading all the necessary libraries: rvest(for web scraping and extracting data from HTML/XML), xml2(For parsing and manipulating XML data), dplyr(for data manipulation and transformation), tidyr( for tidying data).
• xml_data <- read_xml(“path/to/your/codelanguage.xml”): Reads the XML file from the specified path and parses it into an XML object(xml_data).
• Identifying Namespaces: xml_ns() this extracts namespaces from the parsed XML document.
• Extracting Data from XML: using xml_find_all(), xml_text(), xml_attr() to extract data from name, id, difficulty, type etc..
• Combining Data into a Data Frame: it combines all exracted vectors (language_name, language_id, etc.) into a data Frame.
• Cleaning and Transforming the Data: By using dplyr and tidyr we can use their functions to clean them and transform them into particular format as per out need

Conclusion

Reading and Extracting XML data from XML files using rvest has increasingly accessible with the help of libraries like tidyverse, xml2. This provides flexibility and better ways to parse and manipulate XML data, making it easier for everyone to work with structured and complex datasets. It is easy to use because of xml2 package as it simplifies the process of loading and parsing XML documents. Also integrating with tidyverse makes it easier to manipulate data and analyze it easily. The ability to handle complex XML structures with attributes and nested elements allows users to extract precisely the information they need. These tools are versatile enough to handle a wide range of applications either using web data, APIs, or others. This process is efficient in terms of both code and execution. these tools are a valuable addition to the R ecosystem, empowering users to tackle XML data with confidence and ease.