How to find duplicate elements or rows in a Spark DataFrame?
May 19, 2021What are broadcast variables in Spark and when to use them?
May 24, 2021- Apache Pig was created by Yahoo. Apache Hive was created by Facebook. Both tools aimed at hiding the complexities of writing MapReduce jobs.
- Pig is similar to procedural language. Hive is closer to declarative SQL
- Both tools take in instructions or SQL and converts them to MapReduce jobs behind the scenes
- Hive organizes the data as table and partitions and this metadata can be persisted in Hive’s metastore. Pig doesn’t offer any way to persist metadata.
- Hive’s architecture supports JDBC/ODBC clients, authentication, authorization, auditing and logging. Pig’s architecture is not that mature to offer the capabilities offered by Hive.
- Hive is widely integrated with other tools in the Big Data ecosystem like HBase, Spark etc.
Should I choose Pig or Hive if I am starting a Big Data project?
Choose Hive.
Pig is not actively developed anymore. Last release was from June 2017
Hive is widely adopted in the big data space and provides great integration capabilities with other popular tools like Spark.
Big Data In Real World
We are a group of Big Data engineers who are passionate about Big Data and related Big Data technologies. We have designed, developed, deployed and maintained Big Data applications ranging from batch to real time streaming big data platforms. We have seen a wide range of real world big data problems, implemented some innovative and complex (or simple, depending on how you look at it) solutions.
