TLDR: 使用 NiFi 和弹簧引导进行操作
简单的 Apache NiFi 操作仪表板
这是一个正在发展中的工作, 请参与一切都是开源的。俄罗斯和我正在研究一个项目, 以建立一些有用的团队来分析他们的流量, 当前集群状态, 启动和停止流, 并有一个丰富的一看仪表板。
Apache NiFi 和相关工具提供了大量数据, 用于聚合、排序、分类、搜索和最终进行机器学习分析。
有很多工具, 从盒子里出来解决这些问题的部分。Ambari 指标、Grafana 和日志搜索提供了海量的数据和分析能力。您可以在日志搜索中轻松找到所有错误, 并查看 Ambari 指标和 Grafana 中正在发生的情况的尼斯图表。
什么是酷与 Apache NiFi 是有 SitetoSite 的任务, 发送所有的出处, 分析, 指标和操作数据, 你需要它的任何地方。包括到 Apache NiFi!这是监视驱动开发 (MDD)。
在这个概念的小证明工作, 我们抓住一些这些流处理它们在 apache NiFi, 然后将它们存储在 apache Hive 3 表中进行分析。我们可能会将数据推送到 HBase, 用于时间序列的聚合和德鲁伊。我们将看到, 因为这扩大。
还有其他数据访问选项, 包括 NiFi REST api 和 NiFi Python api。
Boostrap通告程序
- NiFi 启动、停止或意外死亡时发送通知
- 两个 OOTB 通知
- 电子邮件通知服务
- HTTP 通知服务
- 编写自定义通知服务很容易
报告任务
- AmbariReportingTask (全局, 每个进程组)
- MonitorDiskUsage (Flowfile、内容、出处库)
- MonitorMemory
监视磁盘使用情况
MonitorActivity
这些是特别有用的事情, 如清除连接。
清除它!
- nipyapi. 画布. purge_connection (con_id)
- nipyapi. 画布. purge_process_group (process_group, 停止 = False)
- nipyapi. 画布. delete_process_group (process_group, 强制 = true, 刷新 = true)
使用案例
指标数据示例
[ {
"appid" : "nifi",
"instanceid" : "7c84501d-d10c-407c-b9f3-1d80e38fe36a",
"hostname" : "princeton1.field.hortonworks.com",
"timestamp" : 1539411679652,
"loadAverage1min" : 0.93,
"availableCores" : 16,
"FlowFilesReceivedLast5Minutes" : 14,
"BytesReceivedLast5Minutes" : 343779,
"FlowFilesSentLast5Minutes" : 0,
"BytesSentLast5Minutes" : 0,
"FlowFilesQueued" : 59952,
"BytesQueued" : 294693938,
"BytesReadLast5Minutes" : 241681,
"BytesWrittenLast5Minutes" : 398753,
"ActiveThreads" : 2,
"TotalTaskDurationSeconds" : 273,
"TotalTaskDurationNanoSeconds" : 273242860763,
"jvmuptime" : 224997,
"jvmheap_used" : 5.15272616E8,
"jvmheap_usage" : 0.9597700387239456,
"jvmnon_heap_usage" : -5.1572632E8,
"jvmthread_statesrunnable" : 11,
"jvmthread_statesblocked" : 2,
"jvmthread_statestimed_waiting" : 26,
"jvmthread_statesterminated" : 0,
"jvmthread_count" : 242,
"jvmdaemon_thread_count" : 125,
"jvmfile_descriptor_usage" : 0.0709,
"jvmgcruns" : null,
"jvmgctime" : null
} ]状态数据示例
{
"statusId" : "a63818fe-dbd2-44b8-af53-eaa27fd9ef05",
"timestampMillis" : "2018-10-18T20:54:38.218Z",
"timestamp" : "2018-10-18T20:54:38.218Z",
"actorHostname" : "princeton1.field.hortonworks.com",
"componentType" : "RootProcessGroup",
"componentName" : "NiFi Flow",
"parentId" : null,
"platform" : "nifi",
"application" : "NiFi Flow",
"componentId" : "7c84501d-d10c-407c-b9f3-1d80e38fe36a",
"activeThreadCount" : 1,
"flowFilesReceived" : 1,
"flowFilesSent" : 0,
"bytesReceived" : 1661,
"bytesSent" : 0,
"queuedCount" : 18,
"bytesRead" : 0,
"bytesWritten" : 1661,
"bytesTransferred" : 16610,
"flowFilesTransferred" : 10,
"inputContentSize" : 0,
"outputContentSize" : 0,
"queuedContentSize" : 623564,
"activeRemotePortCount" : null,
"inactiveRemotePortCount" : null,
"receivedContentSize" : null,
"receivedCount" : null,
"sentContentSize" : null,
"sentCount" : null,
"averageLineageDuration" : null,
"inputBytes" : null,
"inputCount" : 0,
"outputBytes" : null,
"outputCount" : 0,
"sourceId" : null,
"sourceName" : null,
"destinationId" : null,
"destinationName" : null,
"maxQueuedBytes" : null,
"maxQueuedCount" : null,
"queuedBytes" : null,
"backPressureBytesThreshold" : null,
"backPressureObjectThreshold" : null,
"isBackPressureEnabled" : null,
"processorType" : null,
"averageLineageDurationMS" : null,
"flowFilesRemoved" : null,
"invocations" : null,
"processingNanos" : null
}
示例失败数据
[ {
"objectId" : "34c3249c-4a42-41ce-b94e-3563409ad55b",
"platform" : "nifi",
"project" : null,
"bulletinId" : 28321,
"bulletinCategory" : "Log Message",
"bulletinGroupId" : "0b69ea51-7afb-32dd-a7f4-d82b936b37f9",
"bulletinGroupName" : "Monitoring",
"bulletinLevel" : "ERROR",
"bulletinMessage" : "QueryRecord[id=d0258284-69ae-34f6-97df-fa5c82402ef3] Unable to query StandardFlowFileRecord[uuid=cd305393-f55a-40f7-8839-876d35a2ace1,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1539633295746-10, container=default, section=10], offset=95914, length=322846],offset=0,name=783936865185030,size=322846] due to Failed to read next record in stream for StandardFlowFileRecord[uuid=cd305393-f55a-40f7-8839-876d35a2ace1,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1539633295746-10, container=default, section=10], offset=95914, length=322846],offset=0,name=783936865185030,size=322846] due to -40: org.apache.nifi.processor.exception.ProcessException: Failed to read next record in stream for StandardFlowFileRecord[uuid=cd305393-f55a-40f7-8839-876d35a2ace1,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1539633295746-10, container=default, section=10], offset=95914, length=322846],offset=0,name=783936865185030,size=322846] due to -40",
"bulletinNodeAddress" : null,
"bulletinNodeId" : "91ab706b-5d92-454e-bc7a-6911d155fdca",
"bulletinSourceId" : "d0258284-69ae-34f6-97df-fa5c82402ef3",
"bulletinSourceName" : "QueryRecord",
"bulletinSourceType" : "PROCESSOR",
"bulletinTimestamp" : "2018-10-18T20:54:39
