When a consumer gets down:
The possible causes that a consumer stop are when SparkJobs and Yarn application stops working.
1. To Check the SparkHB:
- ps -ef | grep SparkHB
This will tell the existing status of the Spark consumer running on the Application with a PID.
If the SparkHB is not working it will return no job running
This will also stop the running Yarn application
To Start the consumer and yarn application again normally,
- cd /usr/hdp/filegps_release/FILEGPSV42/
2. Troubleshooting of Consumers:
These might be a chance of downtime if the Spark2, Yarn and also the dependent services stop. Then the services needed to be started from the Ambari and check for Status of Spark and Yarn application
Note: Only one Yarn and SparkHB will be running from the FileGPS Application.
- ps -ef | grep Kafka
If Kafka is not running the data will not be pushed to Spark Streaming.
So, to start the Kafka service, go to
- Go to Ambari and restart the Kafka services.
- telnet 10.5.0.X 6667
Port 6667 is the default port for Kafka. When it is deployed as a service for Ambari. If 6667 port is listening by Internal IP of Server where Kafka is Deployed, This ensures that Kafka service is working properly.
Note: Port 6667 can be configurable upon requirment.