March 7, 2023
This is a common problem if you are bringing data from a legacy system or simply from a system which you don’t have control over. First […]
February 25, 2023
www.bigdatainrealworld.com is now fully live Hadoop In Real World is now Big Data In Real World! In-case you missed our communication from last week, you can […]
February 19, 2023
We got an announcement to make. But first, what we have been up to? We have big things planned for 2023 and we have been busy […]
May 11, 2022
Let’s say you have a Hive table and the Hive table is pointing at a location or directory which has several sub directories and each subdirectories […]
April 27, 2022
Let’s say we have a DataFrame like below. +---------+-------+---------------+ | Project| Name|Cost_To_Project| +---------+-------+---------------+ |Ingestion| Jerry| 1000| |Ingestion| Arya| […]
April 20, 2022
Let’s say we have a DataFrame with the employee name, project and the cost of the employee to the project. From this data, we have a […]
April 13, 2022
Deleting a single document is pretty straightforward in Elasticsearch. We can simply issue a DELETE on the document id and the document will be deleted from […]
April 6, 2022
How to fix “could not connect to the endpoint URL” issue in S3? Buckets are tied to regions in Amazon S3. So when we are using […]
March 30, 2022
How to parse information from URL in Hive? Hive offers 2 functions to work with URLS – parse_url and parse_url_tuple. With both functions you can extract […]
March 23, 2022
In this post we are going to describe how to pivot and unpivot a DataFrame in Spark. We have an employee DataFrame with 3 columns, name, […]
March 16, 2022
stack function in Spark takes a number of rows as an argument followed by expressions. stack(n, expr1, expr2.. exprn) stack function will generate n rows by […]
March 9, 2022
Both map and flatMap functions are transformation functions. When applied on RDD, map and flatMap transform each element inside the rdd to something. Consider this simple […]
March 2, 2022
In this post we have shown how to copy an index and all its contents to a new index in Elasticsearch. In this post we are […]
February 23, 2022
In this post we will describe how to copy an index and its contents to a new index in Elasticsearch. We currently have an index named […]
February 16, 2022
You can not directly view or preview the content of a GZip file in HDFS. HDFS does not provide a direct command to do that. Solution […]
February 9, 2022
Simple problem with a simple solution. Solution Use hdfs dfs -count to get the count of files and directories inside the directory. [hirw@wk1 ~]$ hdfs dfs […]
February 2, 2022
Both get_json_object and json_tuple functions in Hive are meant to work with JSON data in Hive. Let’s create a table with some JSON data to work […]
January 26, 2022
Let’s consider the below table employee_depts with 2 columns – ename and dept_list. dept_list is of type array and has the list of departments. CREATE TABLE […]
January 19, 2022
It is quite reasonable for Hive cli or Beeline users to want to see the database they are currently on and it’s frustrating that this is […]
January 12, 2022
The difference between static and dynamic partitioning only exists when the partition is being created based on how the partitions are added to the table. Once […]
