April 5, 2021
By default the messages you can send and manage in Kafka should be less than 1 MB. To increase this limit there are few properties you […]
April 2, 2021
Common problem with a pretty simple solution. Solution When you are doing the directory listing use the -R option to recursively list the directories. If you […]
March 31, 2021
There are several different variations and ways when it comes to inserting or loading data into Hive tables. This post will cover 3 broad ways to […]
March 29, 2021
When you search or lookup a document, Elasticsearch by default returns or shows you all the fields in the document. $ curl -X GET "localhost:9200/account/_doc/954?pretty" { […]
March 26, 2021
cache() and persist() functions are used to cache intermediate results of a RDD or DataFrame or Dataset. You can mark an RDD, DataFrame or Dataset to […]
March 24, 2021
HDFS divides the files into blocks and stores the blocks locally in datanodes. The location varies from cluster to cluster based on the configuration in hdfs-site.xml […]
March 22, 2021
If you are using Kafka on the public cloud like AWS or Azure or on Docker you are more likely to experience the below error. WARN […]
March 19, 2021
Here are three high level reasons using cache() or persist() functions would be appropriate and helpful. Performing multiple actions on the same RDD numbersRDD is used […]
March 17, 2021
In this post we are going to see how to enable version on a bucket and then how to get a very specific version of a […]
March 15, 2021
Both INNER JOIN and LEFT SEMI JOIN return matching records between both tables with a subtle difference. Let’s consider 2 tables – employee and employee_department_mapping with […]
March 12, 2021
Over time we have written several posts on Spark joins and join algorithms explaining the internal working of these join algorithms. Here are all the posts […]
March 10, 2021
Don’t run the command until you fully understand what you are trying to do. hadoop namenode -format will format or delete all files and folders in […]
March 8, 2021
Ever wondered the differences between Rabbit MQ and Kafka and the need for Kafka when there is already RabbitMQ? Read through the post and you will […]
March 5, 2021
Shuffle-and-Replication does not mean a “true” shuffle as in records with the same keys are sent to the same partition. Instead the entire partition of the […]
March 3, 2021
EBS, S3 and Glacier are different storage options available in Amazon Web Services. They differ in cost, use case and purpose. Here is a super quick […]
March 1, 2021
This is a common problem because most of the data files that come from the legacy system will contain a header in the first row. This […]
February 26, 2021
Broadcast Nested Loop join works by broadcasting one of the entire datasets and performing a nested loop to join the data. So essentially every record from […]
February 24, 2021
The answer to this question depends on the version of Hadoop you are using. Older version of Hadoop (1.x.x) The first argument of copyFromLocal is restricted […]
February 22, 2021
Elasticsearch offers 2 different contexts on how you have query or filter documents in an index – Query context and Filter context. Query context In the […]
February 19, 2021
Take a look at the below execution plan. Currently when you print the executed plan, you see that Spark is using Sort Merge Join. scala> dfJoined.queryExecution.executedPlan […]
