Submit Spark Job With Out Uploading Jar in Livy
Introduction to DLI Livy
DLI Livy is an Apache Livy-based client tool used to submit Spark jobs to DLI.
Preparations
- Create a queue. Set Queue Usage to For general purpose, that is, the computing resource of the Spark job. For details, encounter Creating a Queue.
- Prepare a Linux ECS for installing DLI Livy.
- Enable ports 30000 to 32767 and port 8998 on the ECS.
- Install JDK on the ECS. JDK one.8 is recommended. Configure Coffee environment variable JAVA_HOME.
- View the ECS details to obtain its private IP address.
- Use an enhanced datasource connexion to connect the DLI queue to the VPC where the Livy case is located. For details, see Enhanced Datasource Connection.
Downloading and Installing DLI Livy
The software bundle used in the following operations is apache-livy-0.7.2.0100-dli-SNAPSHOT-bin.tar.gz. Replace it with the latest one.
- Download the DLI Livy software package.
- Use WinSCP to upload the obtained software bundle to the prepared ECS directory.
- Log in to ECS as user root and perform the following steps to install DLI Livy:
- Run the following command to create an installation directory:
mkdir Livy installation directory
For example, to create the /opt/livy directory, run the mkdir /opt/livy control. The following operations use the /opt/livy installation directory equally an example. Supplant information technology as required.
- Run the following control to decompress the software package to the installation directory:
tar --extract --file apache-livy-0.7.ii.0100-dli-SNAPSHOT-bin.tar.gz --directory /opt/livy --strip-components 1 --no-same-owner
- Run the post-obit commands to modify the configuration file name:
cd /opt/livy/ conf
mv livy-client.conf.template livy-client.conf
mv livy.conf.template livy.conf
mv livy-env.sh.template livy-env.sh
mv log4j.properties.template log4j.backdrop
mv spark-blacklist.conf.template spark-blacklist.conf
impact spark-defaults.conf
- Run the following command to create an installation directory:
Modifying the DLI Livy Configuration File
- Upload the specified DLI Livy JAR parcel to the OBS saucepan directory.
- Log in to OBS console and create a directory for storing the DLI Livy JAR package in the specified OBS bucket, for case: obs://bucket/livy/jars/.
- Obtain the post-obit JAR packages and upload them to the OBS bucket directory created in one.a:
/opt/livy/rsc-jars/livy-api-0.7.2.0100-dli-SNAPSHOT.jar /opt/livy/rsc-jars/livy-rsc-0.seven.2.0100-dli-SNAPSHOT.jar /opt/livy/repl_2.eleven-jars/livy-core_2.eleven-0.7.2.0100-dli-SNAPSHOT.jar /opt/livy/repl_2.11-jars/livy-repl_2.11-0.7.2.0100-dli-SNAPSHOT.jar
- Modify the DLI Livy configuration file.
- Run the following control to modify the /opt/livy/conf/livy-customer.conf configuration file:
6 /opt/livy/conf/livy-client.conf
Add the following content to the file and modify the configuration items as required:
# Set the private IP accost of the ECS, which can be obtained past running the ifconfig command. livy.rsc.launcher.accost = X.Ten.10.X # Ready the ports enabled on the ECS. livy.rsc.launcher.port.range = 30000~32767
- Run the following command to modify the /opt/livy/conf/livy.conf configuration file:
vi /opt/livy/conf/livy.conf
Add the following content to the file and modify the configuration items as required:
livy.server.port = 8998 livy.spark.chief = yarn livy.server.contextLauncher.custom.class=org.apache.livy.rsc.DliContextLauncher livy.server.batch.custom.course=org.apache.livy.server.batch.DliBatchSession livy.server.interactive.custom.class=org.apache.livy.server.interactive.DliInteractiveSession livy.server.sparkApp.custom.class=org.apache.livy.utils.SparkDliApp livy.server.recovery.mode = recovery livy.server.recovery.state-store = filesystem # Change the following file directory of DLI Livy every bit needed: livy.server.recovery.state-store.url = file:///opt/livy/shop/ livy.server.session.timeout-check = true livy.server.session.timeout = 1800s livy.server.session.state-retain.sec = 1800s livy.dli.spark.version = two.three.2 livy.dli.spark.scala-version = 2.11 # Enter the OBS bucket directory created in 1.a for storing the Livy JAR package. livy.repl.jars = obs://bucket/livy/jars/livy-core_2.eleven-0.7.two.0100-dli-SNAPSHOT.jar, obs://bucket/livy/jars/livy-repl_2.xi-0.7.2.0100-dli-SNAPSHOT.jar livy.rsc.jars = obs://bucket/livy/jars/livy-api-0.vii.ii.0100-dli-SNAPSHOT.jar, obs://bucket/livy/jars/livy-rsc-0.vii.2.0100-dli-SNAPSHOT.jar
- Run the following command to modify the /opt/livy/conf/spark-defaults.conf configuration file:
half dozen /opt/livy/conf/spark-defaults.conf
Add the following parameters to the file. For details well-nigh the parameter configurations, see Table 1.
# The following parameters tin be overwritten when a job is submitted. spark.yarn.isPython=true spark.pyspark.python=python3 # Enter the production environment URL of DLI. spark.dli.user.uiBaseAddress=https://console.huaweicloud.com/dli/web # Set the region where the queue is located. spark.dli.user.regionName=XXXX # Prepare the DLI endpoint accost. spark.dli.user.dliEndPoint=XXXX # Enter the proper name of the created DLI queue. spark.dli.user.queueName=XXXX # Set the AK used for submitting a job. spark.dli.user.admission.primal=XXXX # Ready the SK used for submitting a task. spark.dli.user.hole-and-corner.key=XXXX # Fix the project ID used for submitting a task. spark.dli.user.projectId=XXXX
Table 1 Mandatory parameters in spark-defaults.conf Parameter
Description
spark.dli.user.regionName
Name of the region where the DLI queue is located.
you lot can obtain the region proper name from Regions and Endpoints.
spark.dli.user.dliEndPoint
Endpoint where the DLI queue is located. Yous can obtain the endpoint from Regions and Endpoints.
spark.dli.user.queueName
Queue name.
spark.dli.user.admission.primal
Access key of a user. The user must take the Spark job permissions. For details, see Permissions Management.
For details about how to obtain the AK/SK, run into Obtaining the AK/SK.
spark.dli.user.hole-and-corner.key
spark.dli.user.projectId
Projection ID, which can exist obtained by refering to Obtaining a Projection ID"Obtaining a Project ID".
The following parameters are optional. Set them based on the parameter clarification and site requirements. For details about these parameters, run into Spark Configuration.
Table 2 Optional parameters in spark-defaults.conf Spark Job Parameter
Spark Batch Processing Parameter
Remarks
spark.dli.user.file
file
Not required for connecting to the notebook tool
spark.dli.user.className
class_name
Not required for connecting to the notebook tool
spark.dli.user.scType
sc_type
Same as the native Livy configuration
spark.dli.user.args
args
Same as the native Livy configuration
spark.submit.pyFiles
python_files
Same as the native Livy configuration
spark.files
files
Same as the native Livy configuration
spark.dli.user.modules
modules
-
spark.dli.user.image
image
Custom image used for submitting a job. This parameter is available for container clusters only and is not set past default.
spark.dli.user.autoRecovery
auto_recovery
-
spark.dli.user.maxRetryTimes
max_retry_times
-
spark.dli.user.catalogName
catalog_name
To access metadata, set this parameter to dli.
- Run the following control to modify the /opt/livy/conf/livy-customer.conf configuration file:
Starting DLI Livy
- Run the following command to go to the DLI Livy installation directory:
Example: cd /opt/livy
- Run the following command to starting time DLI Livy:
./bin/livy-server start
Submitting a Spark Job Using DLI Livy
The following shows how to employ the curl command and DLI Livy to submit a Spark job to DLI.
- Upload the JAR file of the adult Spark task program to the OBS directory.
For example, upload spark-examples_2.11-XXXX.jar to the obs://bucket/path directory.
- Log in to the ECS server where DLI Livy is installed as user root.
- Run the ringlet command to submit a Spark job asking to DLI using DLI Livy.
ECS_IP indicates the private IP address of the ECS where DLI Livy is installed.
whorl --location --request Postal service 'http:// ECS_IP :8998/batches' \ --header 'Content-Type: awarding/json' \ --data-raw '{ "driverMemory": "3G", "driverCores": i, "executorMemory": "2G", "executorCores": 1, "numExecutors": ane, "args": [ "1000" ], "file": "obs://bucket/path/spark-examples_2.11-XXXX.jar", "className": "org.apache.spark.examples.SparkPi", "conf": { "spark.dynamicAllocation.minExecutors": ane, "spark.executor.instances": 1, "spark.dynamicAllocation.initialExecutors": 1, "spark.dynamicAllocation.maxExecutors": 2 }
Source: https://support.huaweicloud.com/intl/en-us/devg-dli/dli_09_0201.html
0 Response to "Submit Spark Job With Out Uploading Jar in Livy"
Post a Comment