Hello I use Hortonworks sandbox 2.6, and there was a problem with running jar-nicks written for spark2. Prior to that, I used version 2.3.2 and there the file system was divided into local and hdfs and it was clear (from 100,500 times) how to run (although I only ran the MR tasks, I think the spark would also have no problems b). In the new version, the file system is not broken, and where to put the jarnik is not clear (wherever it is not shoved). And I do not understand how to write a command to launch a jarn. Ash read a bunch of tutorials and a half with different teams (namely, --master to --deploy-mode are different), but for now, because of inexperience, I don’t understand the difference. If there is anyone faced with HDP 2.6 roztolkuyte

  1. where to put jarnik
  2. how to write a command correctly
  3. and look, jarniku like OK ordinary (Ie only my classes) without third-party libraries?

Photo file system in Ambari Photo file system in Ambari

    1 answer 1

    So so who cares who does not. I will write how I managed to run.

    1. Jarnik put in /tmp/spark/*.jar .

    2. Sorcery over the startup team. Here is the command that launched it all:

       spark-submit \ --class movie.Movie \ --deploy-mode cluster \ --master yarn \ --driver-memory 512mb \ --executor-memory 512mb \ --executor-cores 1 \ --num-executors 2 \ hdfs:///tmp/spark/spark.jar

      Errors that made in the team:

      • --driver-memory and --executor-memory not less than 512 MB (for smaller values ​​an exception is generated). If you don’t bet, the default may be big numbers and you simply don’t have enough resources.

    3. Where to look for what the error is if the status of the task is FAILED . To do this, use the command

       yarn logs -applicationId {appId}

      where appId is your job ID. And scroll through a huge log in search of errors. And then google to help.